Environment: HDP 2.3. Oozie v4.2 (Oozie server 4.2.0.2.3.0.0-2557)
Hi,
Oozie job with Spark action workflow fails with the following log:
Error: Could not load YARN classes. This copy of Spark may not have been compiled with YARN support.
The namenode and jobTracker (ResourceManager) address:port have been set correctly.
I am trying out the Spark workflow example provided with Oozie (http://archive.apache.org/dist/oozie/4.2.0/)
As the same Spark job runs successfully with spark-submit, I replaced the contents of the hdfs:///user/oozie/share/lib/lib_20150815235635/spark with the jars in /usr/hdp/current/spark-client/lib. (See http://stackoverflow.com/questions/30904316/oozie-sparkaction-for-yarn-cluster/32108349). After this, the error no longer appears.
Any reason why HDP 2.3 ships with Oozie Spark libs that are not built with YARN support? Also is copying the libs from /usr/hdp/current/spark-client/lib the right way to resolve this?
Regards,
Ram