I am trying the mahout 1.0 on the name node with HDP 2.2 deployed using Ambari 2.0. I am trying to run it locally with specify the -ma parameter. No matter what memory parameter I passed in via the command line, I always get warning. It seems that the parameter did not take effect at all (see –1). The SPARK_HOME is set to /usr/hdp/current/spark-client on the name node. If I pass in “-ma yarn-cluster”, I got nullPointerException (see –2). Does anybody experience this issue and have any solution? Thanks
./mahout spark-itemsimilarity –sparkExecutorMem 20g -m 10 \
-D:spark.driver.memory=8g,spark.executor.memory=4g \
–input /user/spark-itemsimilarity/part-m-00000 \
–output /user/spark-itemsimilarity/result
–1)
15/05/19 13:22:03 WARN MemoryStore: Not enough space to cache rdd_3_3 in memory! (computed 757.1 MB so far)
15/05/19 13:22:03 INFO MemoryStore: Memory use = 1078.9 MB (blocks) + 748.8 MB (scratch space shared across 1 thread(s)) = 1827.7 MB. Storage limit = 1966.1 MB.
–2)
15/05/19 14:06:55 INFO YarnClusterScheduler: Created YarnClusterScheduler
Exception in thread “main” java.lang.NullPointerException
at org.apache.spark.deploy.yarn.ApplicationMaster$.getAttempId(ApplicationMaster.scala:538)
at org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend.start(YarnClusterSchedulerBackend.scala:34)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:140)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:350)
at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:95)
at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:88)
at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201)
at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
at scala.Option.map(Option.scala:145)
at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)