Quantcast
Viewing all articles
Browse latest Browse all 3435

Oozie SPARK job submit to YARN cluster connection refused

I am trying to submit a spark job using OOZIE to YARN cluster on HDP 2.3 configured on amazon instance with a 6 node cluster. Spark job runs fine when I execute using spark-submit having master as yarn-cluster but as soon as I configure OOZIE workflow with same configurations the job gets to RUNNING state in no time but then it waits for about 20 minutes before getting killed. Apparently while checking the job history logs it states the exception is connection refused as follows:

Error: application failed with exception
java.net.ConnectException: Call From ip-instance-ip.ec2.internal:8020/ip-instance-ip.ec2.internal to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
	at org.apache.hadoop.ipc.Client.call(Client.java:1431)
	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:206)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy17.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:506)
	

As shown in the logs it is trying to connect to port 8032 but my job-tracker port is 8050. Following are the NAME NODE and JOB TRACKER URLS that I am using, which by the way works well for SQOOP import jobs, but fails on YARN cluster.

nameNode=hdfs://ip-instance-ip.ec2.internal:8020
jobTracker=ip-instance-ip.ec2.internal:8050

And I have checked in AMBARI the yarn.resourcemanager.address is ip-instance-ip.ec2.internal:8050 which I pass to jobTracker node in OOZIE. Still I don’t know why it goes to port 8032, I know that is default port but HDP is configured on 8050. How can I resolve this issue ?


Viewing all articles
Browse latest Browse all 3435

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>