It appears you haven’t setup the HDP version. See http://hortonworks.com/hadoop-tutorial/using-apache-spark-hdp/
and ensure you have the following
“Create a file SPARK_HOME/conf/spark-defaults.conf and add the following settings:
spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041″