Kafka Cluster 2019(1)ZooKeeper Cluster 3.5.5

Kafka Cluster 2019(1)ZooKeeper Cluster 3.5.5

Set up 2*N+1 Cluster for ZooKeeper first.

Check JAVA Version
> java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

Find the latest version from here
https://www.apache.org/dyn/closer.cgi/zookeeper/

I download the latest version
> wget http://mirrors.ocf.berkeley.edu/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz

Unzip and place in the working directory
> sudo ln -s /home/carl/tool/zookeeper-3.5.5 /opt/zookeeper-3.5.5
> sudo ln -s /opt/zookeeper-3.5.5 /opt/zookeeper

Add this to the path
export PATH="/opt/zookeeper/bin:$PATH"

Prepare the configuration file
> cp conf/zoo_sample.cfg conf/zoo.cfg

If our port 8080 is taken, we need to change AdminServer port from
> vi bin/zkServer.sh
nohup "$JAVA" $ZOO_DATADIR_AUTOCREATE "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.admin.serverPort=8081"\
    "-Dzookeeper.log.file=${ZOO_LOG_FILE}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
    -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -9 %p' \
    -cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

Command to start the service
> zkServer.sh start conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Starting zookeeper ... STARTED

Check the Status from Admin Console
http://ubuntu-master:8081/commands/stats

Connect from the client
> zkCli.sh -server localhost:2181

Stop the Server
> zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

Set Up Cluster Configuration
In the configuration
server.1=ubuntu-master:2888:3888
server.2=ubuntu-dev2:2888:3888
server.3=ubuntu-dev4:2888:3888

> cat /tmp/zookeeper/myid
1

Add this to PATH
ZOO_MY_ID=1
export ZOO_MY_ID

Do the similar things to server2 and server3
PATH="/opt/zookeeper/bin:$PATH"
ZOO_MY_ID=2
export ZOO_MY_ID
> mkdir -p /tmp/zookeeper/
> cat /tmp/zookeeper/myid
2


PATH="/opt/zookeeper/bin:$PATH"
ZOO_MY_ID=3
export ZOO_MY_ID
> mkdir -p /tmp/zookeeper/
> cat /tmp/zookeeper/myid
3

Start all 3 servers
> zkServer.sh start conf/zoo1.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo1.cfg
Starting zookeeper ... STARTED

> zkServer.sh start conf/zoo2.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo2.cfg
Starting zookeeper ... STARTED

> zkServer.sh start conf/zoo3.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo3.cfg
Starting zookeeper ... STARTED

Check status on them
> zkServer.sh status conf/zoo1.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo1.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

> zkServer.sh status conf/zoo2.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo2.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

> zkServer.sh status conf/zoo3.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo3.cfg
Client port found: 2181. Client address: localhost.
Mode: follower


References:
https://medium.com/thinkspecial/power-of-reactor-kafka-5df7cdef2bdd
https://github.com/reactor/reactor-kafka
https://www.e4developer.com/2018/04/11/getting-reactive-with-spring-boot-2-0-and-reactor/
https://www.reactiveprogramming.be/an-introduction-to-reactor-kafka/
https://objectpartners.com/2018/08/07/apache-kafka-and-reactive-spring-boot/
https://github.com/reactor/reactor-kafka
https://projectreactor.io/docs/kafka/release/reference/#_non_blocking_back_pressure

Zookeeper Cluster
https://www.jianshu.com/p/ea138dba2d36