Reply To: Kafka and Spark Streaming – NoSuchMethodError: kafka.consumer.SimpleConsumer.

For example after downloading

spark-streaming-kafka_2.10-1.3.1.2.3.2.0-2650.jar

to my local directory and placing

cluster-log4j.properties

, content below, in the same directory:

og4j.appender.stderr=org.apache.log4j.ConsoleAppender
log4j.appender.stderr.Target=System.err
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
log4j.appender.stderr.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [thread:%t]  %-5p %c{1} - %m%n
log4j.rootLogger=INFO, stderr
log4j.logger.com.streamsets=DEBUG

and executing the following:

spark-submit --master yarn-cluster --executor-memory 1024m --executor-cores 1 --num-executors 1 --files cluster-log4j.properties --conf 'spark.driver.extraClassPath=/usr/hdp/2.3.0.0-2557/kafka/libs/*' --conf 'spark.executor.extraClassPath=/usr/hdp/2.3.0.0-2557/kafka/libs/*' --jars spark-streaming-kafka_2.10-1.3.1.2.3.2.0-2650.jar --conf 'spark.driver.extraJavaOptions=-Dlog4j.debug -Dlog4j.configuration=cluster-log4j.properties' --conf 'spark.executor.extraJavaOptions=-Dlog4j.debug -Dlog4j.configuration=cluster-log4j.properties' --class org.apache.spark.examples.streaming.JavaDirectKafkaWordCount /usr/hdp/2.3.0.0-2557/spark/lib/spark-examples-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar sandbox.hortonworks.com:6667 INPUT

I get the error above.