kafka quick start

Kafka是由LinkedIn开发的一个分布式的消息系统，使用Scala编写

 

kafka是基于zookeeper的，在启动kafka之前要先启动zookeeper，kafka默认自带zookeeper，不需要自带的可以自行安装。

 

1.下载安装

官网下载，解压到 /usr/local下

 

shifulong kafka_2.9.1-0.8.2.2$ pwd
/usr/local/kafka_2.9.1-0.8.2.2
 
shifulong kafka_2.9.1-0.8.2.2$ ll
total 32
drwxr-xr-x@   8 shifulong  staff    272 10 12 10:37 ./
drwxrwxr-x   23 root       admin    782 10 12 10:17 ../
-rw-r--r--@   1 shifulong  staff  11358  9  3 09:34 LICENSE
-rw-r--r--@   1 shifulong  staff    162  9  3 09:34 NOTICE
drwxr-xr-x@  21 shifulong  staff    714  9  3 09:44 bin/
drwxr-xr-x@  12 shifulong  staff    408 10 15 14:16 config/
drwxr-xr-x@  18 shifulong  staff    612  9  3 09:44 libs/
drwxr-xr-x  125 shifulong  staff   4250 10 15 14:03 logs/

 

 

2.启动zookeeper 

shifulong kafka_2.9.1-0.8.2.2$ bin/zookeeper-server-start.sh config/zookeeper.properties 

 

 

3.启动kafka

由于到单机模拟4台kafka，需要启动4个server

config/server.properties 拷贝四分，做如下修改 

broker.id全局唯一，从0开始

 

config/server.properties:  //默认配置
    broker.id=0
    port=9093
    log.dir=/tmp/kafka-logs-1
 
config/server-1.properties:
    broker.id=1
    port=9093
    log.dir=/tmp/kafka-logs-1
 
config/server-2.properties:
    broker.id=2
    port=9094
    log.dir=/tmp/kafka-logs-2
 
config/server-3.properties:
    broker.id=3
    port=9095
    log.dir=/tmp/kafka-logs-3

 

启动4个broker

shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server.properties 
shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server-1.properties 
shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server-2.properties 
shifulong kafka_2.9.1-0.8.2.2$ bin/kafka-server-start.sh config/server-3.properties 

 

 

4.创建topic

• 1.假设实验环境中Kafka集群只有一个broker，log.dirs=/tmp/kafka-logs

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic test_single_broker

执行完创建命令后，在/tmp/kafka-logs 目录下 会自动创建四个partition，以topic名称+编号命名(从0开始)

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:36 test_single_broker-0/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:36 test_single_broker-1/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:36 test_single_broker-2/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:36 test_single_broker-3/

 

• 2.假设实验环境中Kafka集群只有4个broker，

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 5 --topic test_multi_broker

 

/tmp/kafka-logs 目录下

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:43 test_multi_broker-2/

 

/tmp/kafka-logs-1 目录下

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:43 test_multi_broker-3/

 

/tmp/kafka-logs-2 目录下

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:43 test_multi_broker-0/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:43 test_multi_broker-4/

 

/tmp/kafka-logs-3 目录下

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:43 test_multi_broker-1/

上面的测试证明了如下问题

在集群环境中通一个topic的多个partition的均匀的在每个broker上创建，

 

• 3.多broker 并且 replication-factor > 1 的情况

与上面的例子不同的是，--replication-factor 2的参数是2(创建1个副本)

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 5 --topic test_multi_replication_topic

/tmp/kafka-logs

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-2/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-3/

 

/tmp/kafka-logs-1

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-0/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-3/

 

/tmp/kafka-logs-2

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-0/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-1/

 

/tmp/kafka-logs-3

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-1/

drwxr-xr-x   4 shifulong  wheel  136 10 15 14:54 test_multi_repli_topic-2/

可以看到每个partition创建了2此在不同的broker上

并且创建是有一定规律的

有一篇较为详细的博客介绍分配的规则：

Kafka集群partition replication自动分配分析   http://blog.csdn.net/lizhitao/article/details/41778193

 

5.发消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test_multi_replication_topic 

 

6.消费消息

 

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test_multi_replication_topic --from-beginning