Hai guys, i have problem with ambari on process pig check service.
The eror is
<pre class=”stdout” style=”padding: 9.5px; font-family: monospace; font-size: 11px; border-radius: 4px; margin-top: 0px; margin-bottom: 10px; line-height: 14px; word-break: break-all; word-wrap: break-word; white-space: pre-wrap; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: #f5f5f5;”>2015-09-21 07:25:07,808 – ExecuteHadoop[‘dfs -rmr pigsmoke.out passwd; hadoop –config /etc/hadoop/conf dfs -put /etc/passwd passwd ‘] {‘security_enabled’: False, ‘keytab’: [EMPTY], ‘conf_dir’: ‘/etc/hadoop/conf’, ‘try_sleep’: 5, ‘kinit_path_local’: ”, ‘tries’: 3, ‘user’: ‘ambari-qa’, ‘bin_dir’: ‘/usr/hdp/current/hadoop-client/bin’}
2015-09-21 07:25:07,809 – Execute[‘hadoop –config /etc/hadoop/conf dfs -rmr pigsmoke.out passwd; hadoop –config /etc/hadoop/conf dfs -put /etc/passwd passwd ‘] {‘logoutput’: False, ‘path’: [‘/usr/hdp/current/hadoop-client/bin’], ‘tries’: 3, ‘user’: ‘ambari-qa’, ‘try_sleep’: 5}
2015-09-21 07:25:13,756 – File[‘/var/lib/ambari-agent/data/tmp/pigSmoke.sh’] {‘content’: StaticFile(‘pigSmoke.sh’), ‘mode’: 0755}
2015-09-21 07:25:13,757 – Execute[‘pig /var/lib/ambari-agent/data/tmp/pigSmoke.sh’] {‘path’: [‘/usr/hdp/current/pig-client/bin:/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin’], ‘tries’: 3, ‘user’: ‘ambari-qa’, ‘try_sleep’: 5}
2015-09-21 07:25:38,470 – ExecuteHadoop[‘fs -test -e pigsmoke.out’] {‘bin_dir’: ‘/usr/hdp/current/hadoop-client/bin’, ‘user’: ‘ambari-qa’, ‘conf_dir’: ‘/etc/hadoop/conf’}
2015-09-21 07:25:38,471 – Execute[‘hadoop –config /etc/hadoop/conf fs -test -e pigsmoke.out’] {‘logoutput’: False, ‘path’: [‘/usr/hdp/current/hadoop-client/bin’], ‘tries’: 1, ‘user’: ‘ambari-qa’, ‘try_sleep’: 0}
2015-09-21 07:25:41,224 – ExecuteHadoop[‘dfs -rmr pigsmoke.out passwd; hadoop –config /etc/hadoop/conf dfs -put /etc/passwd passwd ‘] {‘security_enabled’: False, ‘keytab’: [EMPTY], ‘conf_dir’: ‘/etc/hadoop/conf’, ‘try_sleep’: 5, ‘kinit_path_local’: ”, ‘tries’: 3, ‘user’: ‘ambari-qa’, ‘bin_dir’: ‘/usr/hdp/current/hadoop-client/bin’}
2015-09-21 07:25:41,225 – Execute[‘hadoop –config /etc/hadoop/conf dfs -rmr pigsmoke.out passwd; hadoop –config /etc/hadoop/conf dfs -put /etc/passwd passwd ‘] {‘logoutput’: False, ‘path’: [‘/usr/hdp/current/hadoop-client/bin’], ‘tries’: 3, ‘user’: ‘ambari-qa’, ‘try_sleep’: 5}
2015-09-21 07:25:46,902 – Could not verify HDP version by calling ‘/usr/bin/hdp-select versions > /tmp/tmpXKfGej’. Return Code: 0, Output: 2.2.8.0-3150
.
2015-09-21 07:25:46,903 – Execute[‘pig -x tez /var/lib/ambari-agent/data/tmp/pigSmoke.sh’] {‘path’: [‘/usr/hdp/current/pig-client/bin:/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin’], ‘tries’: 3, ‘user’: ‘ambari-qa’, ‘try_sleep’: 5}
2015-09-21 07:25:54,987 – Retrying after 5 seconds. Reason: Execution of ‘pig -x tez /var/lib/ambari-agent/data/tmp/pigSmoke.sh’ returned 2. 15/09/21 07:25:48 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/09/21 07:25:48 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/09/21 07:25:48 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
15/09/21 07:25:48 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
15/09/21 07:25:48 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType
2015-09-21 07:25:48,386 [main] INFO org.apache.pig.Main – Apache Pig version 0.14.0.2.2.8.0-3150 (rexported) compiled Sep 09 2015, 21:56:01
2015-09-21 07:25:48,386 [main] INFO org.apache.pig.Main – Logging error messages to: /home/ambari-qa/pig_1442834748385.log
2015-09-21 07:25:49,461 [main] INFO org.apache.pig.impl.util.Utils – Default bootup file /home/ambari-qa/.pigbootup not found
2015-09-21 07:25:49,609 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://streamserver3.detik.com:8020
2015-09-21 07:25:50,852 [main] INFO org.apache.pig.tools.pigstats.ScriptState – Pig features used in the script: UNKNOWN
2015-09-21 07:25:50,888 [main] INFO org.apache.pig.data.SchemaTupleBackend – Key [pig.schematuple] was not set… will not generate code.
2015-09-21 07:25:50,921 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer – {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2015-09-21 07:25:51,123 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher – Tez staging directory is /tmp/temp-483531544
2015-09-21 07:25:51,148 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler – File concatenation threshold: 100 optimistic? false
2015-09-21 07:25:51,251 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers – Generating mapreduce api input splits
2015-09-21 07:25:51,319 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat – Total input paths to process : 1
2015-09-21 07:25:51,319 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1
2015-09-21 07:25:51,361 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths (combined) to process : 1
2015-09-21 07:25:51,380 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers – NumSplits: 1, SerializedSize: 391
2015-09-21 07:25:51,975 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: guava-11.0.2.jar
2015-09-21 07:25:51,976 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: joda-time-2.8.2.jar
2015-09-21 07:25:51,976 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: antlr-runtime-3.4.jar
2015-09-21 07:25:51,976 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: pig-0.14.0.2.2.8.0-3150-core-h2.jar
2015-09-21 07:25:51,976 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: automaton-1.11-8.jar
2015-09-21 07:25:52,171 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.io.sort.mb to 614 from MR setting mapreduce.task.io.sort.mb
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class
2015-09-21 07:25:52,172 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent
2015-09-21 07:25:52,173 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout
2015-09-21 07:25:52,174 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec
2015-09-21 07:25:52,300 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – For vertex – scope-5: parallelism=1, memory=1536, java opts=-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.8.0-3150 -Xmx1229m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA
2015-09-21 07:25:52,524 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState – Pig script settings are added to the job
2015-09-21 07:25:52,748 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient – Tez Client Version: [ component=tez-api, version=0.5.2.2.2.8.0-3150, revision=74d41acebaf7a5ba32c4f7b145cf4f83be39b159, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTIme=20150909-2131 ]
2015-09-21 07:25:53,578 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl – Timeline service address: http://streamserver3.detik.com:8188/ws/v1/timeline/
2015-09-21 07:25:53,682 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy – Connecting to ResourceManager at streamserver3.detik.com/203.190.243.43:8050
2015-09-21 07:25:53,774 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient – Session mode. Starting session.
2015-09-21 07:25:53,775 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils – Using tez.lib.uris value from configuration: /hdp/apps/2.2.8.0-3150/tez/tez.tar.gz
2015-09-21 07:25:53,777 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob – Cannot submit DAG
java.io.FileNotFoundException: File does not exist: /hdp/apps/2.2.8.0-3150/tez/tez.tar.gz
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1219)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:750)
at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:127)
at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:178)
at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:724)
at org.apache.tez.client.TezClient.start(TezClient.java:301)
at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95)
at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159)
at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-09-21 07:25:54,479 [main] INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats – Script Statistics:
HadoopVersion: 2.6.0.2.2.8.0-3150
PigVersion: 0.14.0.2.2.8.0-3150
TezVersion: 0.5.2.2.2.8.0-3150
UserId: ambari-qa
FileName: /var/lib/ambari-agent/data/tmp/pigSmoke.sh
StartedAt: 2015-09-21 07:25:51
FinishedAt: 2015-09-21 07:25:54
Features: UNKNOWN
Failed!
DAG PigLatin:pigSmoke.sh-0_scope-0:
ApplicationId: null
TotalLaunchedTasks: -1
FileBytesRead: -1
FileBytesWritten: -1
HdfsBytesRead: 0
HdfsBytesWritten: 0
Input(s):
Output(s):
2015-09-21 07:25:54,482 [main] ERROR org.apache.pig.tools.grunt.GruntParser – ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/ambari-qa/pig_1442834748385.log
2015-09-21 07:25:54,512 [main] INFO org.apache.pig.Main – Pig script completed in 6 seconds and 232 milliseconds (6232 ms)
2015-09-21 07:25:54,512 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher – Shutting down thread pool
2015-09-21 07:25:54,516 [Thread-1] ERROR org.apache.pig.impl.io.FileLocalizer – java.io.IOException: Filesystem closed
2015-09-21 07:26:07,836 – Retrying after 5 seconds. Reason: Execution of ‘pig -x tez /var/lib/ambari-agent/data/tmp/pigSmoke.sh’ returned 2. 15/09/21 07:26:01 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/09/21 07:26:01 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/09/21 07:26:01 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
15/09/21 07:26:01 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
15/09/21 07:26:01 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType
2015-09-21 07:26:01,503 [main] INFO org.apache.pig.Main – Apache Pig version 0.14.0.2.2.8.0-3150 (rexported) compiled Sep 09 2015, 21:56:01
2015-09-21 07:26:01,503 [main] INFO org.apache.pig.Main – Logging error messages to: /home/ambari-qa/pig_1442834761501.log
2015-09-21 07:26:02,589 [main] INFO org.apache.pig.impl.util.Utils – Default bootup file /home/ambari-qa/.pigbootup not found
2015-09-21 07:26:02,729 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://streamserver3.detik.com:8020
2015-09-21 07:26:03,727 [main] INFO org.apache.pig.tools.pigstats.ScriptState – Pig features used in the script: UNKNOWN
2015-09-21 07:26:03,767 [main] INFO org.apache.pig.data.SchemaTupleBackend – Key [pig.schematuple] was not set… will not generate code.
2015-09-21 07:26:03,801 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer – {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2015-09-21 07:26:03,988 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher – Tez staging directory is /tmp/temp-1903215029
2015-09-21 07:26:04,012 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler – File concatenation threshold: 100 optimistic? false
2015-09-21 07:26:04,103 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers – Generating mapreduce api input splits
2015-09-21 07:26:04,166 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat – Total input paths to process : 1
2015-09-21 07:26:04,166 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1
2015-09-21 07:26:04,204 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths (combined) to process : 1
2015-09-21 07:26:04,222 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers – NumSplits: 1, SerializedSize: 391
2015-09-21 07:26:04,823 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: guava-11.0.2.jar
2015-09-21 07:26:04,823 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: joda-time-2.8.2.jar
2015-09-21 07:26:04,823 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: antlr-runtime-3.4.jar
2015-09-21 07:26:04,823 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: pig-0.14.0.2.2.8.0-3150-core-h2.jar
2015-09-21 07:26:04,823 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: automaton-1.11-8.jar
2015-09-21 07:26:05,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent
2015-09-21 07:26:05,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent
2015-09-21 07:26:05,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.io.sort.mb to 614 from MR setting mapreduce.task.io.sort.mb
2015-09-21 07:26:05,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent
2015-09-21 07:26:05,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor
2015-09-21 07:26:05,021 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies
2015-09-21 07:26:05,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes
2015-09-21 07:26:05,023 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent
2015-09-21 07:26:05,023 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout
2015-09-21 07:26:05,023 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec
2015-09-21 07:26:05,148 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – For vertex – scope-5: parallelism=1, memory=1536, java opts=-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.8.0-3150 -Xmx1229m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA
2015-09-21 07:26:05,369 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState – Pig script settings are added to the job
2015-09-21 07:26:05,597 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient – Tez Client Version: [ component=tez-api, version=0.5.2.2.2.8.0-3150, revision=74d41acebaf7a5ba32c4f7b145cf4f83be39b159, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTIme=20150909-2131 ]
2015-09-21 07:26:06,481 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl – Timeline service address: http://streamserver3.detik.com:8188/ws/v1/timeline/
2015-09-21 07:26:06,587 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy – Connecting to ResourceManager at streamserver3.detik.com/203.190.243.43:8050
2015-09-21 07:26:06,680 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient – Session mode. Starting session.
2015-09-21 07:26:06,680 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils – Using tez.lib.uris value from configuration: /hdp/apps/2.2.8.0-3150/tez/tez.tar.gz
2015-09-21 07:26:06,682 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob – Cannot submit DAG
java.io.FileNotFoundException: File does not exist: /hdp/apps/2.2.8.0-3150/tez/tez.tar.gz
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1219)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:750)
at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:127)
at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:178)
at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:724)
at org.apache.tez.client.TezClient.start(TezClient.java:301)
at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95)
at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159)
at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-09-21 07:26:07,323 [main] INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats – Script Statistics:
HadoopVersion: 2.6.0.2.2.8.0-3150
PigVersion: 0.14.0.2.2.8.0-3150
TezVersion: 0.5.2.2.2.8.0-3150
UserId: ambari-qa
FileName: /var/lib/ambari-agent/data/tmp/pigSmoke.sh
StartedAt: 2015-09-21 07:26:04
FinishedAt: 2015-09-21 07:26:07
Features: UNKNOWN
Failed!
DAG PigLatin:pigSmoke.sh-0_scope-0:
ApplicationId: null
TotalLaunchedTasks: -1
FileBytesRead: -1
FileBytesWritten: -1
HdfsBytesRead: 0
HdfsBytesWritten: 0
Input(s):
Output(s):
2015-09-21 07:26:07,326 [main] ERROR org.apache.pig.tools.grunt.GruntParser – ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/ambari-qa/pig_1442834761501.log
2015-09-21 07:26:07,356 [main] INFO org.apache.pig.Main – Pig script completed in 6 seconds and 26 milliseconds (6026 ms)
2015-09-21 07:26:07,356 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher – Shutting down thread pool
2015-09-21 07:26:07,360 [Thread-1] ERROR org.apache.pig.impl.io.FileLocalizer – java.io.IOException: Filesystem closed
2015-09-21 07:26:21,179 – Error while executing command ‘service_check’:
Traceback (most recent call last):
File “/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py”, line 123, in execute
method(env)
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/PIG/package/scripts/service_check.py”, line 96, in service_check
user = params.smokeuser
File “/usr/lib/python2.6/site-packages/resource_management/core/base.py”, line 148, in __init__
self.env.run()
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 149, in run
self.run_action(resource, action)
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 115, in run_action
provider_action()
File “/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py”, line 241, in action_run
raise ex
Fail: Execution of ‘pig -x tez /var/lib/ambari-agent/data/tmp/pigSmoke.sh’ returned 2. 15/09/21 07:26:14 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/09/21 07:26:14 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/09/21 07:26:14 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
15/09/21 07:26:14 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
15/09/21 07:26:14 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType
2015-09-21 07:26:14,422 [main] INFO org.apache.pig.Main – Apache Pig version 0.14.0.2.2.8.0-3150 (rexported) compiled Sep 09 2015, 21:56:01
2015-09-21 07:26:14,423 [main] INFO org.apache.pig.Main – Logging error messages to: /home/ambari-qa/pig_1442834774421.log
2015-09-21 07:26:15,552 [main] INFO org.apache.pig.impl.util.Utils – Default bootup file /home/ambari-qa/.pigbootup not found
2015-09-21 07:26:15,697 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://streamserver3.detik.com:8020
2015-09-21 07:26:16,703 [main] INFO org.apache.pig.tools.pigstats.ScriptState – Pig features used in the script: UNKNOWN
2015-09-21 07:26:16,750 [main] INFO org.apache.pig.data.SchemaTupleBackend – Key [pig.schematuple] was not set… will not generate code.
2015-09-21 07:26:16,784 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer – {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2015-09-21 07:26:17,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher – Tez staging directory is /tmp/temp744045582
2015-09-21 07:26:17,065 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler – File concatenation threshold: 100 optimistic? false
2015-09-21 07:26:17,237 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers – Generating mapreduce api input splits
2015-09-21 07:26:17,355 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat – Total input paths to process : 1
2015-09-21 07:26:17,355 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1
2015-09-21 07:26:17,425 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths (combined) to process : 1
2015-09-21 07:26:17,458 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers – NumSplits: 1, SerializedSize: 391
2015-09-21 07:26:18,155 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: guava-11.0.2.jar
2015-09-21 07:26:18,155 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: joda-time-2.8.2.jar
2015-09-21 07:26:18,155 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: antlr-runtime-3.4.jar
2015-09-21 07:26:18,155 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: pig-0.14.0.2.2.8.0-3150-core-h2.jar
2015-09-21 07:26:18,155 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – Local resource: automaton-1.11-8.jar
2015-09-21 07:26:18,351 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.io.sort.mb to 614 from MR setting mapreduce.task.io.sort.mb
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records
2015-09-21 07:26:18,352 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout
2015-09-21 07:26:18,353 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper – Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec
2015-09-21 07:26:18,481 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler – For vertex – scope-5: parallelism=1, memory=1536, java opts=-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.8.0-3150 -Xmx1229m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA
2015-09-21 07:26:18,706 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState – Pig script settings are added to the job
2015-09-21 07:26:18,933 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient – Tez Client Version: [ component=tez-api, version=0.5.2.2.2.8.0-3150, revision=74d41acebaf7a5ba32c4f7b145cf4f83be39b159, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTIme=20150909-2131 ]
2015-09-21 07:26:19,671 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl – Timeline service address: http://streamserver3.detik.com:8188/ws/v1/timeline/
2015-09-21 07:26:19,774 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy – Connecting to ResourceManager at streamserver3.detik.com/203.190.243.43:8050
2015-09-21 07:26:19,866 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient – Session mode. Starting session.
2015-09-21 07:26:19,866 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils – Using tez.lib.uris value from configuration: /hdp/apps/2.2.8.0-3150/tez/tez.tar.gz
2015-09-21 07:26:19,869 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob – Cannot submit DAG
java.io.FileNotFoundException: File does not exist: /hdp/apps/2.2.8.0-3150/tez/tez.tar.gz
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1219)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:750)
at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:127)
at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:178)
at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:724)
at org.apache.tez.client.TezClient.start(TezClient.java:301)
at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95)
at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159)
at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-09-21 07:26:20,662 [main] INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats – Script Statistics:
HadoopVersion: 2.6.0.2.2.8.0-3150
PigVersion: 0.14.0.2.2.8.0-3150
TezVersion: 0.5.2.2.2.8.0-3150
UserId: ambari-qa
FileName: /var/lib/ambari-agent/data/tmp/pigSmoke.sh
StartedAt: 2015-09-21 07:26:17
FinishedAt: 2015-09-21 07:26:20
Features: UNKNOWN
Failed!
DAG PigLatin:pigSmoke.sh-0_scope-0:
ApplicationId: null
TotalLaunchedTasks: -1
FileBytesRead: -1
FileBytesWritten: -1
HdfsBytesRead: 0
HdfsBytesWritten: 0
Input(s):
Output(s):
2015-09-21 07:26:20,665 [main] ERROR org.apache.pig.tools.grunt.GruntParser – ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/ambari-qa/pig_1442834774421.log
2015-09-21 07:26:20,695 [main] INFO org.apache.pig.Main – Pig script completed in 6 seconds and 450 milliseconds (6450 ms)
2015-09-21 07:26:20,695 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher – Shutting down thread pool
2015-09-21 07:26:20,699 [Thread-1] ERROR org.apache.pig.impl.io.FileLocalizer – java.io.IOException: Filesystem closed
can anyone helps me ? what i must to do