Hi,
I am trying to set up flume on sandbox to fetch tweets from Twitter, I am not sure if I am using the correct agent for Twitter as a source. The flume doesn’t start at all, it just hangs there.
-The flume.conf looks like:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
#TwitterAgent.sources.Twitter.type = exec
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = QfX3XXXXXXXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.consumerSecret = iuxRtOTOmiMd03kwkcIqXXXXXXXXXXXXXXXXXzUo
TwitterAgent.sources.Twitter.accessToken = 3106XXXXXXXXXXXXXXXXEG2ud7AYr2eprKB2qgyOBB6Z2e1
TwitterAgent.sources.Twitter.accessTokenSecret = k8XXXXXXXXXXXXXXXl2fwQ4W4Aeb
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, bus$
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://192.168.221.128:8020/hdp/apps/2.2.0.0-2041/flume/twitterData
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100
–flume-env.sh:
# Enviroment variables can be set here.
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS=”-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote”
# Note that the Flume conf directory is always included in the classpath.
FLUME_CLASSPATH=”/usr/hdp/2.2.0.0-2041/flume/lib/flume-sources-1.0-SNAPSHOT.jar”
# export HIVE_HOME=/usr/lib/hive
# export HCAT_HOME=/usr/lib/hive-hcatalog
The command to run is:
[root@sandbox flume]# ./bin/flume-ng agent -n TwitterAgent -c conf -f /usr/hdp/2.2.0.0-2041/flume/conf/flume.conf
and then the process just hangs, after :
……………………………………..
…………………………………………
a-6.jar:/usr/hdp/2.2.0.0-2041/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.2.0.0-2041/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.2.0.0-2041/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/conf:/lib/*’ -Djava.library.path=::/usr/hdp/2.2.0.0-2041/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.2.0.0-2041/hadoop/lib/native::/usr/hdp/2.2.0.0-2041/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.2.0.0-2041/hadoop/lib/native org.apache.flume.node.Application -n TwitterAgent -f /usr/hdp/2.2.0.0-2041/flume/conf/flume.conf
Nothing after this.
Could you please tell me if I am using the correct agent for Twitter? Since the one I am using is from cloudera, does Hortonworks has its own implementation of the Twitter Agent?
Please help!