Kafka Cluster 2019(6) 3 Nodes Cluster on CentOS7

Kafka Cluster 2019(6) 3 Nodes Cluster on CentOS7

Here is my machines:
192.168.56.110    rancher-home
192.168.56.111    rancher-worker1
192.168.56.112    rancher-worker2

Need JAVA ENV JDK8, 11, 12
> sudo yum install git
> git clone https://github.com/gcuisinier/jenv.git ~/.jenv
> echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.bash_profile
> echo 'eval "$(jenv init -)"' >> ~/.bash_profile
> . ~/.bash_profile

Check version
> jenv --version
jenv 0.5.2-12-gdcbfd48

Download JDK 8, 11, 12 from Official website
https://www.oracle.com/technetwork/java/javase/downloads/jdk12-downloads-5295953.html
jdk-11.0.4_linux-x64_bin.tar.gz
jdk-12.0.2_linux-x64_bin.tar.gz
jdk-8u221-linux-x64.tar.gz

Unzip all of these files and place in working directory, link to /opt directory
> tar zxvf jdk-11.0.4_linux-x64_bin.tar.gz
> tar zxvf jdk-12.0.2_linux-x64_bin.tar.gz
> tar zxvf jdk-8u221-linux-x64.tar.gz

> mv jdk-11.0.4 ~/tool/
> mv jdk-12.0.2 ~/tool/
> mv jdk1.8.0_221 ~/tool/

> sudo ln -s /home/carl/tool/jdk1.8.0_221 /opt/jdk-1.8.0
> sudo ln -s /home/carl/tool/jdk-11.0.4 /opt/jdk-11.0.4
> sudo ln -s /home/carl/tool/jdk-12.0.2 /opt/jdk-12.0.2

Add to JENV
> jenv add /opt/jdk-1.8.0
> jenv add /opt/jdk-11.0.4
> jenv add /opt/jdk-12.0.2

Check the installed versions
> jenv versions
  11.0
  11.0.4
  12.0
  12.0.2
  1.8
  1.8.0.221
  oracle64-11.0.4
  oracle64-12.0.2
  oracle64-1.8.0.221

Try to set global to 11
> jenv global 11.0

Check the java version
> java -version
java version "11.0.4" 2019-07-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.4+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.4+10-LTS, mixed mode)

Install Latest Zookeeper
Find the latest version from here
https://www.apache.org/dyn/closer.cgi/zookeeper/

I download the latest version
> wget http://mirrors.ocf.berkeley.edu/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz

Unzip and place in the working directory
> tar zxvf apache-zookeeper-3.5.5-bin.tar.gz
> mv apache-zookeeper-3.5.5-bin ~/tool/zookeeper-3.5.5
> sudo ln -s /home/carl/tool/zookeeper-3.5.5 /opt/zookeeper-3.5.5
> sudo ln -s /opt/zookeeper-3.5.5 /opt/zookeeper

Add this to the path
> vi ~/.bash_profile
PATH=$PATH:/opt/zookeeper/bin
ZOO_MY_ID=1
export ZOO_MY_ID
> . ~/.bash_profile

Prepare the configuration for the Zookeeper Cluster

> cd /opt/zookeeper
> mkdir data
> cp conf/zoo_sample.cfg conf/zoo.cfg
> vi conf/zoo.cfg
server.1=rancher-home:2888:3888
server.2=rancher-worker1:2888:3888
server.3=rancher-worker2:2888:3888
dataDir=/opt/zookeeper/data


> vi /opt/zookeeper/data/myid
1

Do the similar to other 2 machines, just change the myid from 1 to 2 or 3.
Start the service on 3 machines
> /opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg

Check the status on all the nodes
> zkServer.sh status conf/zoo.cfg
/home/carl/.jenv/shims/java
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

> zkServer.sh status conf/zoo.cfg
/home/carl/.jenv/shims/java
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

One Server will be the leader, other 2 servers will be the followers.

Find the latest Kafka
https://kafka.apache.org/downloads

My current latest version is 2.12-2.3.0
> wget https://www-us.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz
Unzip and place it in the right directories
> tar zxvf kafka_2.12-2.3.0.tgz
> mv kafka_2.12-2.3.0 ~/tool/
> sudo ln -s /home/carl/tool/kafka_2.12-2.3.0 /opt/kafka-2.3.0
> sudo ln -s /opt/kafka-2.3.0 /opt/kafka

Configure on 3 machines
> cd /opt/kafka
> vi config/server.properties

On the first Node
broker.id=1
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181

On the second Node
broker.id=2
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181

On the Third Node
broker.id=3
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181

Start the Kafka Service on 3 Nodes
> nohup /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &

3 Servers are started successfully.

Verify from Clients
Create One Topic
> bin/kafka-topics.sh --create --zookeeper rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181 --replication-factor 2 --partitions 2 --topic cluster1
Created topic cluster1.

Producer
> bin/kafka-console-producer.sh --broker-list rancher-home:9092,rancher-worker1:9092,rancher-worker2:9092 --topic cluster1
>hello
>sillycat
>yiyikang
>where is the pokemon game?

Consumer
> bin/kafka-console-consumer.sh --bootstrap-server rancher-home:9092,rancher-worker1:9092,rancher-worker2:9092 --topic cluster1 --from-beginning
sillycat
hello
yiyikang
where is the pokemon game?