Reply To: Spark 1.2.1 with HDP 2.2

Incase anyone else needs to perform this these are the steps I took to get it to work with Spark 1.2.1 as well as Spark 1.3.0-RC3:

1. Pull 1.2.1 Source
2. Apply the following patches
a. Address jackson version, https://github.com/apache/spark/pull/3938
b. Address the propagation of the hdp.version set in the spark-default.conf, https://github.com/apache/spark/pull/3409
3. build with $SPARK_HOME./make-distribution.sh –name hadoop2.6 –tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests package

Then deploy the resulting artifact => spark-1.2.1-bin-hadoop2.6.tgz following instructions in the HDP Spark preview http://hortonworks.com/hadoop-tutorial/using-apache-spark-hdp/

FWIW spark-1.3.0-RC3 appears to be working fine with HDP as well and steps 2a and 2b are not required, just the make-distribution.

-Todd