0

我正在从事一项与 Hadoop 和水槽相关的小任务。我正在尝试使用水槽获取 Twitter 数据,但出现以下错误

错误 node.AbstractConfigurationProvider:由于配置过程中的错误,已删除 Sink HDFS java.lang.InstantiationException:定义了不兼容的接收器和通道设置。sink 的批量大小大于通道事务容量。Sink:HDFS,批量大小 = 1000,通道 MemChannel,事务容量 = 100

flume_twitter.conf

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = XXXXXXXXX
TwitterAgent.sources.Twitter.consumerSecret = XXXXXXXXXX
TwitterAgent.sources.Twitter.accessToken = XXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXXXXXXXXXXXX
TwitterAgent.sources.Twitter.keywords = RioCompetition, cpfc, Palace, London, Christmas, New Years

TwitterAgent.sinks.HDFS.type = hdfs 
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/TwitterData
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream 
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 100
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 
TwitterAgent.sinks.HDFS.hdfs.rollCount = 1000 
 
# Describing/Configuring the channel 
TwitterAgent.channels.MemChannel.type = memory 
TwitterAgent.channels.MemChannel.capacity = 100
TwitterAgent.channels.MemChannel.transactionCapacity = 100
  
# Binding the source and sink to the channel 
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel 

水槽-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

FLUME_CLASSPATH=/home/mauli/Downloads/flume-sources-1.0-SNAPSHOT.jar

下面的命令用于运行文件

flume-ng 代理 –conf ./conf/ -f ../conf/flume_twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent

我也尝试过以下命令

flume-ng 代理 TwitterAgent -f ../conf/flume_twitter.conf

我已经尝试了很多方法,几乎​​所有堆栈溢出的引用都
可以帮助我解决这个错误
提前谢谢!

4

0 回答 0