Data Solution 2019(2)Kafka Single on Docker

Data Solution 2019(2)Kafka Single on Docker

Download the binary
> wget http://apache.osuosl.org/kafka/2.1.0/kafka_2.11-2.1.0.tgz
Unzip the file and place in the working directory
Start the single Zookeeper
> bin/zookeeper-server-start.sh config/zookeeper.properties
Start the Kafka Server
> bin/kafka-server-start.sh config/server.properties
Verify my installation, create a topic testyiyi
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testyiyi
Send some messages in producer
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testyiyi
Start a consumer to receive these informations
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testyiyi --from-beginning
Try to run that in Docker
This is the start.sh, in the shell script I need to wait until the zookeeper starts
#!/bin/sh -ex
#start the service
cd /tool/kafka
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > /tmp/zookeeper.log &
sleep 1
while ! grep -m1 'binding to port' < /tmp/zookeeper.log; do
    sleep 1
done
bin/kafka-server-start.sh config/server.properties
Makefile is almost the same, nothing special
IMAGE=sillycat/public
TAG=ubuntu-kafka-1.0
NAME=ubuntu-kafka-1.0
prepare:
    wget http://apache.osuosl.org/kafka/2.1.0/kafka_2.11-2.1.0.tgz -P install/
docker-context:
build: docker-context
    docker build -t $(IMAGE):$(TAG) .
run:
    docker run -d -p 2181:2181 -p 9092:9092 --name $(NAME) $(IMAGE):$(TAG)
debug:
    docker run -ti -p 2181:2181 -p 9092:9092 --name $(NAME) $(IMAGE):$(TAG) /bin/bash
clean:
    docker stop ${NAME}
    docker rm ${NAME}
logs:
    docker logs ${NAME}
publish:
    docker push ${IMAGE}
Dockerfile, it installed the oracle JDK, that is nice
#Run a kafka server side
#Prepare the OS
FROM            ubuntu:16.04
MAINTAINER      Carl Luo <[e-mail]>
ENV DEBIAN_FRONTEND noninteractive
ENV JAVA_HOME       /usr/lib/jvm/java-8-oracle
ENV LANG            en_US.UTF-8
ENV LC_ALL          en_US.UTF-8
RUN apt-get -qq update
RUN apt-get -qqy dist-upgrade
#Prepare the denpendencies
RUN apt-get install -qy wget unzip
#Install SUN JAVA
RUN apt-get update && \
  apt-get install -y --no-install-recommends locales && \
  locale-gen en_US.UTF-8 && \
  apt-get dist-upgrade -y && \
  apt-get --purge remove openjdk* && \
  echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections && \
  echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" > /etc/apt/sources.list.d/webupd8team-java-trusty.list && \
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 && \
  apt-get update && \
  apt-get install -y --no-install-recommends oracle-java8-installer oracle-java8-set-default && \
  apt-get clean all
#add the software
RUN             mkdir /tool/
RUN             mkdir /admin/
WORKDIR         /tool/
ADD             install/kafka_2.11-2.1.0.tgz /tool/
RUN             ln -s /tool/kafka_2.11-2.1.0 /tool/kafka
ADD             conf/server.properties /tool/kafka/config
#set up the app
EXPOSE  2181 9092
RUN     mkdir -p /app/
ADD     start.sh /app/
WORKDIR /app/
CMD    [ "./start.sh" ]
Most of the content from server.properties are good, I only change the listeners and advertised.listerners
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://localhost:9092
References:
https://kafka.apache.org/quickstart
Wait for server to start in shell script
https://stackoverflow.com/questions/21475639/wait-until-service-starts-in-bash-script
Install JDK 8
https://github.com/mlaccetti/docker-oracle-java8-ubuntu-16.04/blob/master/Dockerfile
https://github.com/wurstmeister/kafka-docker/wiki/Connectivity