Hi All,
I have configured cross realm among 2 secured clusters.
hadoop fs -ls hdfs://remotenn:8020/ —–>able to see files.
hadoop fs -cp hdfs://source/path hdfs://dest/path worked from client only as hadoop.namenode.kerberos.pattern -* enabled only here.
If I try to run above 2 commands they run only if namenode.kerberos.pattern is enabled in hdfs-site.xml.
a)Shall I set this parameter through out the cluster?
b)Do i need to set dfs.socket.timeout and dfs.datanode.socket.write.timeout to some value.Currently write is set to 0 and read (socket.timeout) not set.
ERROR WHILE DISTCP COMMAND with current set up.
when I do hadoop distcp —>
It fails with below error.I have checked dfs.socket.timeout and dfs.datanode.socket.write.timeout in hdfs-site.xml ->both set to 0
15/08/31 19:02:57 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW
15/08/31 19:02:59 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW
15/08/31 19:03:00 INFO impl.YarnClientImpl: Submitted application application_1440506170094_2185
15/08/31 19:03:00 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/bdauser/.staging/job_1440506170094_2185
15/08/31 19:03:00 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Call From gvlhdmpap02/39.7.56.2 to 10.234.22.56:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/39.7.56.2:42716 remote=10.234.22.56/10.234.22.56:8020]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:375)
bdauser@gvlengpap01:~>
IT guys say no firewall there among prod and nonprod.
Please suggest.
Regards,
Rajat