flume-ng安装

 

下载flume-ng

$ wget http://mirrors.hust.edu.cn/apache/flume/1.5.2/apache-flume-1.5.2-bin.tar.gz

解压，创建文件输出目录

创建flume的配置

$ vi conf/flume-conf.properties

 

agent1.sources = source1

agent1.sinks = sink1

agent1.channels = channel1

 

# Describe/configure source1

agent1.sources.source1.type = avro

agent1.sources.source1.bind = 10.111.131.68 # avro监听host

agent1.sources.source1.port = 44446 # avro监听port

agent1.sources.source1.channels = channel1

 

# Use a channel which buffers events in memory

agent1.channels.channel1.type = memory

agent1.channels.channel1.capacity = 1000

agent1.channels.channel1.transactionCapactiy = 100 

 

# Bind the source and sink to the channel

# Describe sink1

agent1.sinks.sink1.type = file_roll # 文件输出

agent1.sinks.sink1.sink.directory = /home/hai/app-monitoring/flume/apache-flume-1.5.2-bin/flumed-logs # 文件输出目录

agent1.sinks.sink1.channel = channel1

 

拷贝

apache-flume-1.5.2-bin/tools/flume-ng-log4jappender-1.5.2-jar-with-dependencies.jar

到需要输出日志的应用的WEB-INF/lib下

修改log4j配置

log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender

log4j.appender.flume.layout=org.apache.log4j.PatternLayout

log4j.appender.flume.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}%5p %F %L %m%n

log4j.appender.flume.Hostname=10.111.131.68

log4j.appender.flume.Port=44446

log4j.rootCategory=DEBUG, console, flume

启动flume

bin/flume-ng agent -n agent1 -c conf -f conf/flume-conf.properties -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

接下来就可以在日志输出目录查看到日志文件了

这个页面可以看到flume的计量信息

http://10.111.131.68:34545/metrics

$ ls flumed-logs/

1426141638517-1   1426141638517-14  1426141638517-19  1426141638517-23  1426141638517-28  1426141638517-32  1426141638517-37  1426141638517-7

1426141638517-10  1426141638517-15  1426141638517-2   1426141638517-24  1426141638517-29  1426141638517-33  1426141638517-38  1426141638517-8

1426141638517-11  1426141638517-16  1426141638517-20  1426141638517-25  1426141638517-3   1426141638517-34  1426141638517-4   1426141638517-9