Flume: compile source code and install

1. download source code

#git clone https://git-wip-us.apache.org/repos/asf/flume.git

 

2. compile

#export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"

#mvn clean install -DskipTests

an error comes

 

see:

https://issues.apache.org/jira/browse/FLUME-2184

 

solution:

#mvn clean install -Dhadoop.profile=2  -DskipTests

 

#mvn clean install -Dhadoop.profile=2  -DskipTests -Dmaven.test.skip=true

 

the last para will compile test classes.(In some situtation, test clasess have some errors)

 

If you want to change the hadoop version, alter the pom.xml

<hadoop2.version>2.6.0</hadoop2.version>

 

3.run

#cd flume-ng-dist/target/apache-flume-1.6.0-SNAPSHOT-bin

#cp conf/flume-conf.properties.template conf/flume.conf

#cp conf/flume-env.sh.template conf/flume-env.sh

Copy and paste this into conf/flume.conf:

 

# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
 
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
 
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = logger
 
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1

 

 #bin/flume-ng agent --conf ./conf/ -f conf/flume.conf

-Dflume.root.logger=DEBUG,console -n agent1

 

 #bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd -Dflume.root.logger=DEBUG,console

 

 

Error: the avro-client  can't work(not read file and sent data to avro souce). when i shutdown the flume agent in other console, the avro-client has a error shows

2014-12-31 11:33:30,865 DEBUG [org.apache.avro.ipc.NettyTransceiver] - Remote peer dmining05/127.0.0.1:41414 closed connection.
2014-12-31 11:33:30,865 DEBUG [org.apache.avro.ipc.NettyTransceiver] - Disconnecting from dmining05/127.0.0.1:41414
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.specific.SpecificData.getClassLoader()Ljava/lang/ClassLoader;
	at org.apache.avro.ipc.specific.SpecificRequestor.getClient(SpecificRequestor.java:158)
	at org.apache.avro.ipc.specific.SpecificRequestor.getClient(SpecificRequestor.java:148)
	at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:171)
	at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
	at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
	at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:170)
	at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:198)
	at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:72)

 

 Solution: the reason is that avro version, replace avro-1.7.4.jar      avro-ipc-1.7.4.jar   avro-mapred-1.7.4.jar  in lib dir using   avro-1.7.7.jar avro-ipc-1.7.7.jar   avro-mapred-1.7.7.jar

 

 

 

-----------

Setting up Eclipse

 

mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs

 

Once this command completes successfully, you must add $HOME/.m2/repository to the classpath in preferences and then you can import all the flume modules as interdependent projects into Eclipse by going to File > Import > General > Existing Projects into Workspace.

 

 

 

 

 

 

 

 

 

 

References

https://cwiki.apache.org/confluence/display/FLUME/Getting+Started

https://cwiki.apache.org/confluence/display/FLUME/Development+Environment