I am also having the same error on an HDP 2.2 that was upgraded from 2.1 running in AWS (Redhat OS). MR, Hive, YARN, etc all worked fine before the upgrade.
The exact error in the YARN logs is: Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
I found this thread that mentioned the YARN classpath issue: http://hortonworks.com/community/forums/topic/strange-mapred-site-xml-after-installing-of-hdp2-2/
And I did notice that my YARN classpath was missing the paths to map-reduce client libs. I added those in. It now looks like:
/etc/hadoop/conf,$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/*,/usr/hdp/current/hadoop-mapreduce-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-app-2.6.0.2.2.0.0-2041.jar
The last entry is actually an exact path to the class its looking for. Restarted MR, YARN, and Hive (I’m trying to run a hive query and getting this error), still no dice. Same error.
Not sure if its related, but I had to fix another upgrade issue related to map-reduce classpath. I set the HADOOP_OPTS var to use the actual version as others have had issue with the variable. It is now like this in mapred_env.sh: export HADOOP_OPTS=”-Dhdp.version=2.2.0.0-2041 $HADOOP_OPTS”
My mapreduce application classpath is now:
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,/usr/hdp/current/hadoop-client/mapreduce.tar.gz
And my mapreduce application framework path is: /hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz
I removed the #mr-framework because of errors saying that framework didn’t exist. This fixed that error.
Any help is appreciated.