Kafka Cluster 2019(2)Kafka Cluster 2.12-2.2.0

Kafka Cluster 2019(2)Kafka Cluster 2.12-2.2.0

Last time, I get the source codes and compile and package it myself. This time, I will just download the binary directly.
> wget http://mirror.cc.columbia.edu/pub/software/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz

Unzip and place it in the right directly
> sudo ln -s /home/carl/tool/kafka-2.12-2.2.0 /opt/kafka-2.2.0
> sudo ln -s /opt/kafka-2.2.0 /opt/kafka

Prepare single file first
> vi config/server.properties
zookeeper.connect=ubuntu-master:2181,ubuntu-dev2:2181,ubuntu-dev4:2181

Add Kafka to path
export PATH="/opt/kafka/bin:$PATH"

Start the Server to Have a Try
> kafka-server-start.sh config/server.properties

Exception:
[2019-06-24 18:25:42,264] INFO Opening socket connection to server ubuntu-dev2/192.168.56.3:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-06-24 18:25:42,265] INFO Socket error occurred: ubuntu-dev2/192.168.56.3:2181: Connection refused (org.apache.zookeeper.ClientCnxn)

Solution:
First of all, I found the Zookeeper cluster is not running.
I checked the logging in ZooKeeper, it is saying that the /tmp/zookeeper/myid file is missing.

Haha, it is because that my /tmp/zookeeper is gone, it is not a good data directory.
Change the zookeeper configuration
dataDir=/opt/zookeeper/data

It works perfect for ZooKeeper now. Try Kafka again.
>kafka-server-start.sh config/server.properties

For single node, it works this time. Try Set Up Cluster.

Do the similar installation on ubuntu-dev2 and ubuntu-dev4.
Copy the basic configuration files here
> scp ubuntu-master:/opt/kafka/config/server1.properties ./config/

Rename them on different machines
> mv config/server1.properties config/server2.properties

> mv config/server1.properties config/server3.properties

Change the broker.id on each configurations files
broker.id=1
broker.id=2
broker.id=3

Clean the log directory and then start the Kafka again.
> rm -fr /tmp/kafka-logs/*

> nohup bin/kafka-server-start.sh config/server1.properties &

> nohup bin/kafka-server-start.sh config/server2.properties &

> nohup bin/kafka-server-start.sh config/server3.properties &

Check the installation.
Create the topic
>bin/kafka-topics.sh --create --zookeeper ubuntu-master:2181,ubuntu-dev2:2181,ubuntu-dev4:2181 --replication-factor 2 --partitions 2 --topic cluster1
Created topic "cluster1".

Producer
>bin/kafka-console-producer.sh --broker-list ubuntu-master:9092,ubuntu-dev2:9092,ubuntu-dev4:9092 --topic cluster1

Consumer
>bin/kafka-console-consumer.sh --zookeeper ubuntu-master:2181,ubuntu-dev2:2181,ubuntu-dev4:2181 --topic cluster1 --from-beginning

Exception:
zookeeper is not a recognized option

Solution:
>bin/kafka-console-consumer.sh --bootstrap-server ubuntu-master:9092,ubuntu-dev2:9092,ubuntu-dev4:9092 --topic cluster1 --from-beginning


It should be working well, next step is to set up driver to connect to it.

References:
https://www.jianshu.com/p/5297773fcc1b