Hello,
today i install kafka, storm and tez 0.5.3 for hadoop 2.6.0 on my cluster, manually
if i understand well, tez would increase performance processing of hive and pig task
I get tez project, i make compile and deliver on my hadoop and i set TEZ configuration.
I execute tez sample with log result below
For Hive , it seem a little bit faster, for pig, i dont soe so much difference thant before. i miss something ?
If somebody have benchmark result about that with old version vs new version with tez ? i am interessting
just for have an idea.
Thanks
JP
seem working with tez. i logs yarn i saw tez container using
hadoop jar ./tez-dist/target/tez-0.5.3/tez-examples-0.5.3.jar orderedwordcount /tests/tez-examples/in /tests/tez-examples/out
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/tez-0.5.3-incubating/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/05/12 21:28:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
hduser@stargate:~/apache-tez-0.5.3-src$ hadoop jar ./tez-dist/target/tez-0.5.3/tez-examples-0.5.3.jar orderedwordcount /tests/tez-examples/in /tests/tez-examples/out
Running OrderedWordCount
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/tez-0.5.3-incubating/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/05/12 21:28:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
15/05/12 21:28:42 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.5.3, revision=${buildNumber}, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150512-1933 ]
15/05/12 21:28:42 INFO client.RMProxy: Connecting to ResourceManager at stargate/192.168.0.11:8032
15/05/12 21:28:43 INFO client.TezClient: Submitting DAG application with id: application_1431457599870_0012
15/05/12 21:28:43 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
15/05/12 21:28:43 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: hdfs://192.168.0.11:9000/apps/tez-0.5.3-incubating,hdfs://192.168.0.11:9000/apps/tez-0.5.3-incubating/lib/
15/05/12 21:28:43 INFO client.TezClient: Tez system stage directory hdfs://192.168.0.11:9000/tmp/hduser/tez/staging/.tez/application_1431457599870_0012 doesn’t exist and is created
15/05/12 21:28:44 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1431457599870_0012, dagName=OrderedWordCount
15/05/12 21:28:44 INFO impl.YarnClientImpl: Submitted application application_1431457599870_0012
15/05/12 21:28:44 INFO client.TezClient: The url to track the Tez AM: http://stargate:8088/proxy/applicati…57599870_0012/
15/05/12 21:28:44 INFO client.RMProxy: Connecting to ResourceManager at stargate/192.168.0.11:8032
15/05/12 21:28:44 INFO client.DAGClientImpl: Waiting for DAG to start running
15/05/12 21:28:52 INFO client.DAGClientImpl: DAG initialized: CurrentState=Running
15/05/12 21:28:53 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:53 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:53 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:54 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 0% TotalTasks: 2 Succeeded: 0 Running: 1 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 0 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 1 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 50% TotalTasks: 2 Succeeded: 1 Running: 1 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 0 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
15/05/12 21:28:59 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 1 Failed: 0 Killed: 0
15/05/12 21:29:00 INFO client.DAGClientImpl: DAG: State: SUCCEEDED Progress: 100% TotalTasks: 2 Succeeded: 2 Running: 0 Failed: 0 Killed: 0
15/05/12 21:29:00 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 0 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
15/05/12 21:29:00 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
15/05/12 21:29:00 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
15/05/12 21:29:00 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED