Hello,
I am running a mapreduce job which internally uses PiG and I am using it for dedup process . It works perfectly fine as I have ran successfully many times for inputs of million rows. Now I am using 2 billion records as a input and started running mapreduce job, it fails in final reduce steps with below error.
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 255
attempt_1418075527007_2636_r_000655_1 100.00 FAILED reduce > reduce dinfhdp07.wellcare.com:8042 logs Sat, 03 Jan 2015 08:59:42 GMT Sat, 03 Jan 2015 12:44:43 GMT 3hrs, 45mins, 1sec AttemptID:attempt_1418075527007_2636_r_000655_1 Timed out after 1000 secs Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143
attempt_1418075527007_2636_r_000656_0 100.00 FAILED reduce > reduce xxxx.xxxx.com:8042 logs Sat, 03 Jan 2015 05:15:23 GMT Sat, 03 Jan 2015 15:08:59 GMT 9hrs, 53mins, 36sec Error: GC overhead limit exceeded Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143.
I am using following settings
SET hive.exec.compress.output=true; SET mapreduce.output.fileoutputformat.compress=true; SET mapreduce.output.fileoutputformat.compress.type=BLOCK; SET mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;SET hive.exec.max.dynamic.partitions.pernode=5555; SET mapreduce.job.reduces=5; SET mapred.reduce.tasks=5; SET hive.exec.max.dynamic.partitions=5555;
SET default_parallel 0; SET mapreduce.job.reduces 0; SET mapreduce.task.timeout 1000000;
Task Type Total Complete
Map 6470 6470
Reduce 867 865
Attempt Type Failed Killed Successful
Maps 0 0 6470
Reduces 4 2 865
counter information:
GC time elapsed (ms) 1344654 231634 1576288
Input split bytes 527116674 0 527116674
Map input records 2535338499 0 2535338499
Map output bytes 989378975223 0 989378975223
Map output materialized bytes 999553986159 0 999553986159
Map output records 2535338499 0