Kafka(5)Upgrade the Version with Multiple Machine

Kafka(5)Upgrade the Version with Multiple Machine

1. Run Single Broker on Single Node
Download the latest version of codes
http://psg.mtu.edu/pub/apache/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz

Unzip the codes and build it.
>./gradlew -PscalaVersion=2.10.0 releaseTarGz -x signArchives

Find the binary file from here
kafka-0.8.1.1/core/build/distributions

Unzip the file and place it in working directory.

Check and change the conf/server.properties

Before we start the kafka, start the zookeeper first
>zkServer.sh start conf/zoo-cluster.cfg

Start the kafka broker
>bin/kafka-server-start.sh config/server.properties

Create the topic
>bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic console-test

Open the sample producer
>bin/kafka-console-producer.sh --broker-list localhost:9092 --topic console-test

2. Run multiple Brokers on Single Machine
Copy and change 

config/server1.properties
broker.id=1
port=9092
log.dirs=/tmp/kafka-logs1

config/server2.properties
broker.id=2
port=9093
log.dirs=/tmp/kafka-logs2

Start the 2 servers
>bin/kafka-server-start.sh config/server1.properties
>bin/kafka-server-start.sh config/server2.properties

Create a Topic on the Cluster
>/opt/kafka$ bin/kafka-topics.sh --create --zookeeper ubuntu-client1:2181,ubuntu-client2:2181,ubuntu-client3:2181 --replication-factor 2 --partitions 2 --topic cluster1

Open the producer Console
>bin/kafka-console-producer.sh --broker-list ubuntu-client1:9092,ubuntu-client1:9093 --topic cluster1

Open the consumer Console to display the message
>bin/kafka-console-consumer.sh --zookeeper ubuntu-client1:2181,ubuntu-client2:2181,ubuntu-client3:2181 --topic cluster1 --from-beginning

3. Run multiple brokers on multiple machines
broker id should be unique, port number should be different on the same machine, so the log dir. That is it.


Create the topic
>bin/kafka-topics.sh --create --zookeeper ubuntu-client1:2181,ubuntu-client2:2181,ubuntu-client3:2181 --replication-factor 2 --partitions 2 --topic cluster2

Producer
>bin/kafka-console-producer.sh --broker-list ubuntu-client1:9092,ubuntu-client2:9092,ubuntu-client3:9092 --topic cluster2

Consumer
>bin/kafka-console-consumer.sh --zookeeper ubuntu-client1:2181,ubuntu-client2:2181,ubuntu-client3:2181 --topic cluster2 --from-beginning



References:
http://www.iteblog.com/archives/1044
http://www.iteblog.com/archives/1045
http://www.iteblog.com/archives/1049