Hi,
I installed HDP 2.2.4 and have copied zlib1.dll to {HADOOP_HOME}/bin. When I run the sample Wordcount problem.It fails with this error
15/05/16 18:57:45 INFO client.RMProxy: Connecting to ResourceManager at <url>/10.128.1.4:8032
15/05/16 18:57:47 INFO input.FileInputFormat: Total input paths to process : 1
15/05/16 18:57:47 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
15/05/16 18:57:47 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 7a4b57bedce694048432dd5bf5b90a6c8ccdba80]
15/05/16 18:57:48 INFO mapreduce.JobSubmitter: number of splits:1
15/05/16 18:57:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1431802381254_0001
15/05/16 18:57:49 INFO impl.YarnClientImpl: Submitted application application_1431802381254_0001
15/05/16 18:57:49 INFO mapreduce.Job: The url to track the job: <url>:8088/proxy/application_1431802381254_0001/
15/05/16 18:57:49 INFO mapreduce.Job: Running job: job_1431802381254_0001
15/05/16 18:58:06 INFO mapreduce.Job: Job job_1431802381254_0001 running in uber mode : false
15/05/16 18:58:06 INFO mapreduce.Job: map 0% reduce 0%
15/05/16 18:58:20 INFO mapreduce.Job: map 100% reduce 0%
15/05/16 18:58:29 INFO mapreduce.Job: Task Id : attempt_1431802381254_0001_r_000000_0, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#15
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:357)
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:279)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:351)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
Now,When I go into Application Master Logs and dig into logs I find that reducer is expecting a gzip file but the mapper output wasn’t
2015-05-16 18:45:20,849 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1431791182314_0011_m_000000_0 decomp: 725274 len: 725278 to MEMORY
2015-05-16 18:45:20,864 WARN [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to shuffle output of attempt_1431791182314_0011_m_000000_0 from VMHadoopDN03.corp.3dmxconsulting.co.uk:13562
java.io.IOException: not a gzip file
at org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.processBasicHeader(BuiltInGzipDecompressor.java:496)
at org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeHeaderState(BuiltInGzipDecompressor.java:257)
at org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:186)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:97)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:534)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
When I got mapper log I see
2015-05-16 18:45:08,532 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output
2015-05-16 18:45:08,532 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2601881; bufvoid = 209715200
2015-05-16 18:45:08,532 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 52428796(209715184); kvend = 51356896(205427584); length = 1071901/13107200
2015-05-16 18:45:09,532 WARN [main] org.apache.hadoop.io.compress.zlib.ZlibFactory: Failed to load/initialize native-zlib library
2015-05-16 18:45:09,532 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.gz]
2015-05-16 18:45:09,532 WARN [main] org.apache.hadoop.mapred.IFile: Could not obtain compressor from CodecPool
2015-05-16 18:45:10,328 INFO [main] org.apache.hadoop.mapred.MapTask: Finished spill 0
I have the configurations in files to use GZIP for all mapred jobs. So on instinct I tried the same job without compression and behold -It works !!
Now the issue I have is how to debug why mapper is not able to produce gzip outputs.
I tried all options.Copying new zlib file from website,copying to to system32 and System64/WOW folders,restarting the cluster but nothing works