I installed HDP 2.2 on Windows Server 2012 R2 Build 9600.
I struggled in executing C# mapper/reducer with configuration file and external DLL. My reducer needs to call external configuration file and also external DLL.
I tried the following
attempt 1)
hadoop jar c:\hdp\hadoop-2.6.0.2.2.0.0-2041\share\hadoop\tools\lib\hadoop-streaming-2.6.0.2.2.0.0-2041.jar -conf hdfs://host_name:8020/user/hadoop/App/Reducer.exe.config -input /user/hadoop/Input -output /user/hadoop/Output -mapper Mapper.exe -reducer Reducer.exe -file C:\Test\Mapper.exe -file C:\Test\Reducer.exe
mapper runs, but reducer fails. The log file reads
==============================================================================================================================================
2015-03-17 17:17:15,788 FATAL [IPC Server handler 0 on 55054] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1426543033330_0007_r_000000_0 – exited : java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2015-03-17 17:17:15,788 INFO [IPC Server handler 0 on 55054] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1426543033330_0007_r_000000_0: Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2015-03-17 17:17:15,815 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1426543033330_0007_r_000000_0: Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
========================================================================================================
attempt 2)
hadoop jar c:\hdp\hadoop-2.6.0.2.2.0.0-2041\share\hadoop\tools\lib\hadoop-streaming-2.6.0.2.2.0.0-2041.jar -conf /user/hadoop/App/Reducer.exe.config -input /user/hadoop/Input -output /user/hadoop/Output -mapper Mapper.exe -reducer Reducer.exe -file C:\Test\Mapper.exe -file C:\Test\Reducer.exe
Mapper runs, but reducer fails with the same error message.
I am wondering what is the correct way/syntax to include the configuration file in the streaming job.
Thanks.