Hi All,
I have a cluster where the hdfs client is firewalled from the rest of the Hadoop cluster. I have been able to secure most access however I am getting errors when monitoring a job. It appears the nodemanager uses a dynamic port each time making it difficult to secure the firewall. Is there a way to restrict the range of ports that are selected to bind the tracking URL to?
The output I receive is as follows:
15/02/11 08:15:22 INFO fs.TestDFSIO: TestDFSIO.1.7
15/02/11 08:15:22 INFO fs.TestDFSIO: nrFiles = 500
15/02/11 08:15:22 INFO fs.TestDFSIO: nrBytes (MB) = 1000.0
15/02/11 08:15:22 INFO fs.TestDFSIO: bufferSize = 1000000
15/02/11 08:15:22 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
15/02/11 08:15:23 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 500 files
15/02/11 08:15:36 INFO fs.TestDFSIO: created control files for: 500 files
15/02/11 08:15:37 INFO impl.TimelineClientImpl: Timeline service address: http://0.0.0.0:8188/ws/v1/timeline/
15/02/11 08:15:37 INFO client.RMProxy: Connecting to ResourceManager at node10.test.local/192.168.0.10:8050
15/02/11 08:15:37 INFO impl.TimelineClientImpl: Timeline service address: http://0.0.0.0:8188/ws/v1/timeline/
15/02/11 08:15:37 INFO client.RMProxy: Connecting to ResourceManager at node10.test.local/192.168.0.10:8050
15/02/11 08:15:38 INFO mapred.FileInputFormat: Total input paths to process : 500
15/02/11 08:15:38 INFO mapreduce.JobSubmitter: number of splits:500
15/02/11 08:15:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1423324247753_0011
15/02/11 08:15:38 INFO impl.YarnClientImpl: Submitted application application_1423324247753_0011
15/02/11 08:15:38 INFO mapreduce.Job: The url to track the job: http://node10.test.lo…24247753_0011/
15/02/11 08:15:38 INFO mapreduce.Job: Running job: job_1423324247753_0011
15/02/11 08:16:03 INFO ipc.Client: Retrying connect to server: node1.test.local/10.10.10.17:33463. Already tried 0 time(s); maxRetries=3
15/02/11 08:16:23 INFO ipc.Client: Retrying connect to server: node1.test.local/10.10.10.17:33463. Already tried 1 time(s); maxRetries=3
15/02/11 08:16:43 INFO ipc.Client: Retrying connect to server: node1.test.local/10.10.10.17:33463. Already tried 2 time(s); maxRetries=3
node1.test.local runs the Ambari DATANODE and NODEMANAGER roles.
The distro is Hortonworks 2.1.1. Please note I have seen this page http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_reference/content/reference_chap2.html but it doesn’t seem to make any reference to such a port range.
Any help will be much appreciated.
Thanks
Charlie
