基于Hadoo的日志收集框架---Chukwa的安装部署

chukwa是解决在集群环境中收集各节点增量日志的一种基于hadoop的实现方案，其主要有如下四个组成部分。
1. Agents 运行在每个客户端上，负责发送数据。
2. Collectors 接收Agents发送的数据并写入稳定存储。
3. MapReduce jobs 分析和归档数据。
4. HICC 数据中心，用于显示数据的web界面。
它的系统架构如下图

以单机部署为例，前提已经安装hadoop(0.20.2)，HADOOP_HOME=/usr/local/cloud/hadoop

1. 下载chukwa(0.4.0)
2. 解压到/usr/local/cloud/chukwa/，CHUKWA_HOME=/usr/local/cloud/chukwa
3. 修改$CHUKWA_HOME/conf/agents，部署在本机的话为localhost
4. 修改$CHUKWA_HOME/conf/collectors， 部署在本机的话为localhost
5. 修改$CHUKWA_HOME/conf/chukwa-env.sh，设置如下项

export JAVA_HOME=/usr/java/jdk1.6.0_26
export HADOOP_HOME="/usr/local/cloud/hadoop"
export HADOOP_CONF_DIR="/usr/local/cloud/hadoop/conf"
export HADOOP_JAR=${HADOOP_HOME}/hadoop-core-0.20.2.jar
export chukwaRecordsRepository="/chukwa/repos/"
export CHUKWA_PID_DIR=/usr/local/cloud/tmp/chukwa/pids
export CHUKWA_LOG_DIR=/usr/local/cloud/tmp/chukwa/logs
export CHUKWA_DATA_DIR=/usr/local/cloud/tmp/chukwa/data

6. 修改$CHUKWA_HOME/conf/chukwa-agent-conf.xml

  <property>
    <name>chukwaAgent.tags</name>
    <value>cluster="chukwa"</value>
    <description>The cluster's name for this agent</description>
  </property>
  
  <property>
    <name>chukwaAgent.control.port</name>
    <value>9093</value>
    <description>The socket port number the agent's control interface can be contacted at.</description>
  </property>

  <property>
    <name>chukwaAgent.hostname</name>
    <value>localhost</value>
    <description>The hostname of the agent on this node. Usually localhost, this is used by the chukwa instrumentation agent-control interface library</description>
  </property>

  <property>
    <name>chukwaAgent.checkpoint.name</name>
    <value>chukwa_agent_checkpoint</value>
    <description>the prefix to to prepend to the agent's checkpoint file(s)</description>
  </property>
  
  <property>
    <name>chukwaAgent.checkpoint.dir</name>
    <value>${CHUKWA_LOG_DIR}/</value>
    <description>the location to put the agent's checkpoint file(s)</description>
  </property>

  <property>
    <name>chukwaAgent.checkpoint.interval</name>
    <value>5000</value>
    <description>the frequency interval for the agent to do checkpoints, in milliseconds</description>
  </property>

  <property>
    <name>chukwaAgent.sender.fastRetries</name>
    <value>4</value>
    <description>the number of post attempts to make to a single collector, before marking it failed</description>
  </property>

  <property>
    <name>chukwaAgent.collector.retries</name>
    <value>144000</value>
    <description>the number of attempts to find a working collector</description>
  </property>

  <property>
    <name>chukwaAgent.collector.retryInterval</name>
    <value>20000</value>
    <description>the number of milliseconds to wait between searches for a collector</description>
  </property>

7. 修改$CHUKWA_HOME/conf/chukwa-collector-conf.xml

 <property>
    <name>writer.hdfs.filesystem</name>
    <value>hdfs://master:9000/</value>
    <description>HDFS to dump to</description>
  </property>

  <property>
    <name>chukwaCollector.outputDir</name>
    <value>/chukwa/logs/</value>
    <description>Chukwa data sink directory</description>
  </property>

  <property>
    <name>chukwaCollector.rotateInterval</name>
    <value>300000</value>
    <description>Chukwa rotate interval (ms)</description>
  </property>

  <property>
    <name>chukwaCollector.http.port</name>
    <value>8080</value>
    <description>The HTTP port number the collector will listen on</description>
  </property>

8. 修改$CHUKWA_HOME/conf/initial_adaptors，添加要监测的日志文件，比如

add filetailer.CharFileTailingAdaptorUTF8 TestLog1 0 /home/matrix/Program/project/log/testlog1 0

9. 启动collectors

$CHUKWA_HOME/bin/start-collectors.sh

10. 启动agents

$CHUKWA_HOME/bin/start-agents.sh

11. 启动data-processors

$CHUKWA_HOME/bin/start-data-processors.sh