had same issue with spark 1.4, solved by following step #2 in the HPD/Spark article http://hortonworks.com/hadoop-tutorial/using-apache-spark-hdp/
make sure use the correct hdp version, that can be seem at /home/hdp if you are on linux, on windows it is usually in the suffix of the folder
Create a file SPARK_HOME/conf/spark-defaults.conf and add the following settings:
spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0–2041
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0–2041