Data Solution 2019(3)Run Zeppelin in Single Docker

Data Solution 2019(3)Run Zeppelin in Single Docker

Exception when Start HDFS in Docker
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Solution:
Add this to ENV solve the problem.
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
Exception when Start HDFS in Docker
Starting namenodes on [0.0.0.0]
0.0.0.0: /tool/hadoop-3.2.0/bin/../libexec/hadoop-functions.sh: line 982: ssh: command not found
Starting datanodes
localhost: /tool/hadoop-3.2.0/bin/../libexec/hadoop-functions.sh: line 982: ssh: command not found
Starting secondary namenodes [140815a59b06]
140815a59b06: /tool/hadoop-3.2.0/bin/../libexec/hadoop-functions.sh: line 982: ssh: command not found
Solution:
https://stackoverflow.com/questions/40801417/installing-ssh-in-the-docker-containers
Install and Start SSH Server
RUN apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN ssh-keygen -q -t rsa -N '' -f /root/.ssh/id_rsa
RUN cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
#start ssh service
nohup /usr/sbin/sshd -D >/dev/stdout &
Exception when Start HDFS
ERROR: JAVA_HOME is not set and could not be found
Solution:
Add JAVA_HOME in Hadoop-env.sh
export JAVA_HOME="/usr/lib/jvm/java-8-oracle”
It seems HDFS is running fine in Docker.
But from the UI, I get error like this from UI http://localhost:9870/dfshealth.html#tab-overview
Exception:
Permission denied: user=dr.who, access=WRITE, inode="/":root:supergroup:drwxr-xr-x
Solution:
https://stackoverflow.com/questions/11593374/permission-denied-at-hdfs
Since this is my local Docker, I will just disable the permission in pdfs-site.xml
    <property>
          <name>dfs.permissions</name>
          <value>false</value>
    </property>
Check Docker Stats
> docker stats
My memory is only 2G, too small, maybe CPU is not power enough as well.
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
382b064708ec        ubuntu-spark-1.0    0.64%               1.442GiB / 1.952GiB   73.89%              216kB / 437kB       255MB / 10.1MB      256
> nproc
4
Maybe CPU is ok
I am using MAC, so the way to increase the memory is to open the tool
Docker Desktop —> References —> Advanced —> CPUs 4, Memory 2GB, Swap 1.0GB
https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container
Clean up my Docker Images which I am not using anymore
> docker images | grep none | awk '{ print $3; }' | xargs docker rmi
Official Website
https://hub.docker.com/r/apache/zeppelin/dockerfile
Finally I made it working.
conf/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://0.0.0.0:9000</value>
    </property>
</configuration>
conf/hadoop-env.sh
export JAVA_HOME="/usr/lib/jvm/java-8-oracle”
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
case ${HADOOP_OS_TYPE} in
  Darwin*)
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.kdc= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf= "
  ;;
esac
conf/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
          <name>dfs.permissions</name>
          <value>false</value>
    </property>
</configuration>
conf/spark-env.sh
HADOOP_CONF_DIR=/tool/hadoop/etc/hadoop
Need to put out zeppelin/conf and zeppelin/notebook out side and mapping to docker application to save data.
This is the important Dockerfile
#Run a kafka server side
#Prepare the OS
FROM            ubuntu:16.04
MAINTAINER      Carl Luo <[e-mail]>
ENV DEBIAN_FRONTEND noninteractive
ENV JAVA_HOME       /usr/lib/jvm/java-8-oracle
ENV LANG            en_US.UTF-8
ENV LC_ALL          en_US.UTF-8
RUN apt-get -qq update
RUN apt-get -qqy dist-upgrade
#Prepare the denpendencies
RUN apt-get install -qy wget unzip vim
RUN apt-get install -qy iputils-ping
#Install SUN JAVA
RUN apt-get update && \
  apt-get install -y --no-install-recommends locales && \
  locale-gen en_US.UTF-8 && \
  apt-get dist-upgrade -y && \
  apt-get --purge remove openjdk* && \
  echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections && \
  echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" > /etc/apt/sources.list.d/webupd8team-java-trusty.list && \
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 && \
  apt-get update && \
  apt-get install -y --no-install-recommends oracle-java8-installer oracle-java8-set-default && \
  apt-get clean all
#Prepare for hadoop and spark
RUN apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN ssh-keygen -q -t rsa -N '' -f /root/.ssh/id_rsa
RUN cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
RUN            mkdir /tool/
WORKDIR        /tool/
#add the software hadoop
ADD            install/hadoop-3.2.0.tar.gz /tool/
RUN            ln -s /tool/hadoop-3.2.0 /tool/hadoop
ADD            conf/core-site.xml /tool/hadoop/etc/hadoop/
ADD            conf/hdfs-site.xml /tool/hadoop/etc/hadoop/
ADD            conf/hadoop-env.sh /tool/hadoop/etc/hadoop/
#add the software spark
ADD            install/spark-2.4.0-bin-hadoop2.7.tgz /tool/
RUN            ln -s /tool/spark-2.4.0-bin-hadoop2.7 /tool/spark
ADD            conf/spark-env.sh /tool/spark/conf/
#add the software zeppelin
ADD            install/zeppelin-0.8.1-bin-all.tgz /tool/
RUN            ln -s /tool/zeppelin-0.8.1-bin-all /tool/zeppelin
#set up the app
EXPOSE  9000 9870 8080 4040
RUN     mkdir -p /app/
ADD     start.sh /app/
WORKDIR /app/
CMD    [ "./start.sh” ]
This is the Makefile which will make it working
IMAGE=sillycat/public
TAG=ubuntu-spark-1.0
NAME=ubuntu-spark-1.0
prepare:
    wget http://mirror.olnevhost.net/pub/apache/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz -P install/
    wget http://ftp.wayne.edu/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz -P install/
    wget http://apache.claz.org/zeppelin/zeppelin-0.8.1/zeppelin-0.8.1-bin-all.tgz -P install/
docker-context:
build: docker-context
    docker build -t $(IMAGE):$(TAG) .
run:
    docker run -d -p 9870:9870 -p 9000:9000 -p 8080:8080 -p 4040:4040 -v $(shell pwd)/zeppelin/notebook:/tool/zeppelin/notebook -v $(shell pwd)/zeppelin/conf:/tool/zeppelin/conf --name $(NAME) $(IMAGE):$(TAG)
debug:
    docker run -ti -p 9870:9870 -p 9000:9000 -p 8080:8080 -p 4040:4040 -v $(shell pwd)/zeppelin/notebook:/tool/zeppelin/notebook -v $(shell pwd)/zeppelin/conf:/tool/zeppelin/conf --name $(NAME) $(IMAGE):$(TAG) /bin/bash
clean:
    docker stop ${NAME}
    docker rm ${NAME}
logs:
    docker logs ${NAME}
publish:
    docker push ${IMAGE}
This is the start.sh to start the application
#!/bin/sh -ex
#prepare ENV
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
export SPARK_HOME="/tool/spark"
#start ssh service
nohup /usr/sbin/sshd -D >/dev/stdout &
#start the service
cd /tool/hadoop
bin/hdfs namenode -format
sbin/start-dfs.sh
cd /tool/zeppelin
bin/zeppelin.sh
After that, we can visit this 3 UI to work on our data
### Hadoop 3.2.0 Spark 2.4.0 Zeppelin 0.8.1
### HDFS
http://localhost:9870/explorer.html#/
### Zeppelin UI
http://localhost:8080/
### After you Run the First Demo JOB, Spark Jobs UI
http://localhost:4040/stages/

References:
https://stackoverflow.com/questions/48129029/hdfs-namenode-user-hdfs-datanode-user-hdfs-secondarynamenode-user-not-defined
https://www.cnblogs.com/sylar5/p/9169090.html
https://www.jianshu.com/p/b49712bbe044
https://stackoverflow.com/questions/40801417/installing-ssh-in-the-docker-containers
https://stackoverflow.com/questions/27504187/ssh-key-generation-using-dockerfile
https://github.com/twang2218/docker-zeppelin