KafKa简单集群环境搭建

KafKa集群配置：

        1.下载解压最新的kafka安装文件 ，并分别解压到三台服务器上      

tar -xzf kafka_2.10-0.8.2.0.tgz
cd kafka_2.10-0.8.2.0

 

        2.修改Server.properties配置文件：

           

  zookeeper.connect=archtest01:2181,archtest02:2181,arch_open:2181

              修改SVR71服务器broker.id=0

              修改SVR72服务器broker.id=1

              修改SVR87服务器broker.id=2

 

       3.启动KafKa服务

     

       mkdir logs

             bin/kafka-server-start.sh config/server.properties > logs/kafka.log&

             zk:文件存储目录：/data/server/zookeeper/data

             KafKa文件存储目录：/tmp

 

       4.测试集群是否安装成功

          

  测试KafKa集群是否安装成功：

     创建Topic
     bin/kafka-topics.sh --create --zookeeper archtest01:2181 --replication-factor 3 --partitions 1 --topic qeeka-topic
     查看Topic
     bin/kafka-topics.sh --describe --zookeeper archtest01:2181 --topic qeeka-topic 

     生产者发送消息
     bin/kafka-console-producer.sh --broker-list archtest01:9092 --topic qeeka-topic
     消费者接收消息
     bin/kafka-console-consumer.sh --zookeeper archtest01:2181 --from-beginning --topic qeeka-topic

 

 

 

   5.简单压力测试(服务器配置较低，需更换服务器后进一步测试，简单测试结果如下：)  

--------------------------------------------------------------------------
生产者：4线程发送50万数据 耗时27秒 每秒1.8万 写入速度每秒17M
[root@archtest02 /usr/local/kafka/kafka_2.10-0.8.2.1]# bin/kafka-producer-perf-test.sh --messages 500000 --message-size 1000 --batch-size 1000 --topics qeeka-topic --threads 4 --broker-list archtest01:9092,archtest02:9092,arch_open:9092
start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2015-03-13 16:46:36:804, 2015-03-13 16:47:03:305, 0, 1000, 1000, 476.84, 17.9932, 500000, 18867.2126
--------------------------------------------------------------------------
消费者：3线程接收50万数据 耗时10秒 每秒1万 读取速度每秒93M
[root@archtest02 /usr/local/kafka/kafka_2.10-0.8.2.1]# bin/kafka-consumer-perf-test.sh --zookeeper archtest01,archtest02,arch_open --messages 500000 --topic qeeka-topic --threads 3
start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2015-03-13 16:47:49:620, 2015-03-13 16:47:59:591, 1048576, 467.1689, 93.9788, 500000, 100583.3836
--------------------------------------------------------------------------
生产者：4线程发送500万数据 耗时247秒 每秒2万 写入速度每秒19M
[root@archtest02 /usr/local/kafka/kafka_2.10-0.8.2.1]# bin/kafka-producer-perf-test.sh --messages 5000000 --message-size 1000 --batch-size 1000 --topics qeeka-topic --threads 4 --broker-list archtest01:9092,archtest02:9092,arch_open:9092
start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2015-03-13 16:55:24:082, 2015-03-13 16:59:31:992, 0, 1000, 1000, 4768.37, 19.2343, 5000000, 20168.6096
--------------------------------------------------------------------------
消费者：3线程接收500万数据 耗时120秒 每秒4.3万 读取速度每秒41M
[root@archtest02 /usr/local/kafka/kafka_2.10-0.8.2.1]# bin/kafka-consumer-perf-test.sh --zookeeper archtest01,archtest02,arch_open --messages 5000000 --topic qeeka-topic --threads 3
start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2015-03-13 17:01:52:511, 2015-03-13 17:03:51:895, 1048576, 4758.7033, 41.6029, 5000000, 43712.4073