Kafka 2017 Update(1)Zookeeper Cluster and Kafka Cluster

Kafka 2017 Update(1)Zookeeper Cluster and Kafka Cluster

Install Zookeeper Cluster on 3 Nodes
>wget http://mirrors.advancedhosters.com/apache/zookeeper/zookeeper-3.5.3-beta/zookeeper-3.5.3-beta.tar.gz
>tar xf zookeeper-3.5.3-beta.tar.gz
>sudo ln -s /home/ec2-user/tool/zookeeper-3.5.3 /opt/zookeeper-3.5.3
>sudo ln -s /opt/zookeeper-3.5.3 /opt/zookeeper

Add the working directory to the PATH
PATH=$PATH:/opt/zookeeper/bin

Prepare the Configuration file
>cp conf/zoo_sample.cfg conf/zoo.cfg
Try start local server
>zkServer.sh start conf/zoo.cfg

Port 8080 is used by the AdminServer which is new in 3.5.0.
Add the settings in the zkServer.sh
    nohup "$JAVA" $ZOO_DATADIR_AUTOCREATE "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.admin.serverPort=8081"\
    "-Dzookeeper.log.file=${ZOO_LOG_FILE}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
    -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -9 %p' \
    -cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &
    if [ $? -eq 0 ]

https://zookeeper.apache.org/doc/r3.5.1-alpha/zookeeperAdmin.html#sc_adminserver_config
Source code is here https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/admin/JettyAdminServer.java
It is reading from the JVM system configuration of -Dzookeeper.admin.serverPort=8081

Visit the Admin console
http://fr-stage-api:8081/commands/stats

Connect with Client
>zkCli.sh -server localhost:2181

Stop the Service
>zkServer.sh stop

Prepare the Configuration for Cluster zoo1.cfg, zoo2.cfg, zoo3.cfg, adding these lines
server.1=fr-stage-api:2888:3888
server.2=fr-stage-consumer:2888:3888
server.3=fr-perf1:2888:3888

>vi /tmp/zookeeper/myid
1
>vi /tmp/zookeeper/myid
2
>vi /tmp/zookeeper/myid
3

Copy the file to all other server and start them all.
Exception:
2018-01-02 19:38:03,370 [myid:] - ERROR [main:QuorumPeerMain@86] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing conf/zoo1.cfg
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:138)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:110)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Caused by: java.lang.IllegalArgumentException: myid file is missing

Solution:
https://github.com/31z4/zookeeper-docker/issues/13
On the system settings:
ZOO_MY_ID=1
export ZOO_MY_ID

This is not necessary if I directly install zookeeper on my local.

Check status on Server
>zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

Kafka Cluster
Kafka requires Gradle 2.0 or higher
Install Gradle Manually https://gradle.org/install/#manually
>wget https://downloads.gradle.org/distributions/gradle-4.4.1-bin.zip
>unzip gradle-4.4.1-bin.zip

>sudo ln -s /home/ec2-user/tool/gradle-4.4.1 /opt/gradle-4.4.1
>sudo ln -s /opt/gradle-4.4.1 /opt/gradle

>gradle --version
------------------------------------------------------------
Gradle 4.4.1
------------------------------------------------------------
Build time:   2017-12-20 15:45:23 UTC
Revision:     10ed9dc355dc39f6307cc98fbd8cea314bdd381c
Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_60 (Oracle Corporation 25.60-b23)
OS:           Linux 4.1.13-18.26.amzn1.x86_64 amd64

Run gradle command on the source directory of Kafka
>gradle

Build Jar package
>./gradlew jar

Build the release Jar Package
>./gradlew releaseTarGz -x signArchives

Copy the binary file out to install directory
>cp ./core/build/distributions/kafka_2.11-1.0.0.tgz ~/install/

Link that from the right working directory
>sudo ln -s /home/ec2-user/tool/kafka-1.0.0 /opt/kafka-1.0.0

Prepare Single Kafka Configuration
>cat config/server.properties
zookeeper.connect=fr-stage-api:2181,fr-stage-consumer:2181,fr-perf1:2181

Start the Server
>kafka-server-start.sh config/server.properties

Prepare the Cluster Configuration
>cp config/server.properties config/server1.properties
>cp config/server.properties config/server2.properties
>cp config/server.properties config/server3.properties
broker.id=1
broker.id=2
broker.id=3

Start the first kafka on the first machine
>nohup bin/kafka-server-start.sh config/server1.properties &

Exceptions:
[2018-01-03 18:17:02,882] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured broker.id 1 doesn't match stored broker.id 0 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getBrokerIdAndOfflineDirs(KafkaServer.scala:615)
at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:92)
at kafka.Kafka.main(Kafka.scala)

Solution:
Clean the log directory
>rm -fr /tmp/kafka-logs/*

>nohup bin/kafka-server-start.sh config/server1.properties &
>nohup bin/kafka-server-start.sh config/server2.properties &
>nohup bin/kafka-server-start.sh config/server3.properties &

Create the topic
>bin/kafka-topics.sh --create --zookeeper fr-stage-api:2181,fr-stage-consumer:2181,fr-perf1:2181 --replication-factor 2 --partitions 2 --topic cluster1
Created topic "cluster1".

Producer
>bin/kafka-console-producer.sh --broker-list fr-stage-api:9092,fr-stage-consumer:9092,fr-perf1:9092 --topic cluster1

Consumer
>bin/kafka-console-consumer.sh --zookeeper fr-stage-api:2181,fr-stage-consumer:2181,fr-perf1:2181 --topic cluster1 --from-beginning


References:
Kafka 1~6
http://sillycat.iteye.com/blog/1563312
http://sillycat.iteye.com/blog/1563314
http://sillycat.iteye.com/blog/2015175
http://sillycat.iteye.com/blog/2015181
http://sillycat.iteye.com/blog/2094688
http://sillycat.iteye.com/blog/2108042

http://sillycat.iteye.com/blog/2215237
http://sillycat.iteye.com/blog/2183932

https://kafka.apache.org/downloads

zookeeper
http://sillycat.iteye.com/blog/2397642
http://sillycat.iteye.com/blog/2397645