Quantcast
Channel: Hortonworks » All Replies
Viewing all articles
Browse latest Browse all 3435

Whether support Spark on YARN, under the HDP in Kerberos environment?

$
0
0

Whether support Spark on YARN, under the HDP in Kerberos environment?
I have 1 Mater +1 Worker
when I run
./bin/spark-submit –class org.apache.spark.examples.SparkPi –master yarn-cluster –num-executors 3 –driver-memory 512m –executor-memory 512m –executor-cores 1 lib/spark-examples*.jar 10
————————————————————————————
result is
14/10/16 10:14:32 INFO yarn.Client: Uploading file:/home/spark/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://mycluster/user/spark/.sparkStaging/application_1413347552991_0018/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar
14/10/16 10:14:33 INFO yarn.Client: Setting up the launch environment
14/10/16 10:14:33 INFO yarn.Client: Setting up container launch context
14/10/16 10:14:33 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx512m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.history.kerberos.enabled=\”true\”, -Dspark.history.kerberos.keytab=\”/etc/security/keytabs/spark.service.keytab”\”, -Dspark.app.name=\”org.apache.spark.examples.SparkPi\”, -Dspark.eventLog.enabled=\”true\”, -Dspark.eventLog.dir=\”file:///home/spark/eventLog\”, -Dspark.history.kerberos.principal=\”spark/hdpsecuremaster.cht.local@CHT.COM.TW\”, -Dspark.history.ui.acls.enable=\”true\”, -Dspark.yarn.historyServer.address=\”hdpsecuremaster.cht.local:18080\”, -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ApplicationMaster, –class, org.apache.spark.examples.SparkPi, –jar , file:/home/spark/lib/spark-examples-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar, –args ’10’ , –executor-memory, 512, –executor-cores, 1, –num-executors , 3, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/10/16 10:14:34 INFO yarn.Client: Submitting application to ASM
Exception in thread “main” java.net.ConnectException: Connection refused…
————————————————————————————
and YARN log like below
2014-10-16 10:14:25,953 INFO ipc.Server (Server.java:saslProcess(1294)) – Auth successful for spark/hdpsecuremaster.cht.local@CHT.COM.TW (auth:SIMPLE)
2014-10-16 10:14:26,114 INFO authorize.ServiceAuthorizationManager (ServiceAuthorizationManager.java:authorize(114)) – Authorization successful for spark/hdpsecuremaster.cht.local@CHT.COM.TW (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
2014-10-16 10:14:26,462 INFO resourcemanager.ClientRMService (ClientRMService.java:getNewApplicationId(249)) – Allocated new applicationId: 18


Viewing all articles
Browse latest Browse all 3435

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>