Data Solution 2019(9)CentOS Installation

Data Solution 2019(9)CentOS Installation

Join two DataFrame and get the count

import org.apache.spark.sql.functions._

val addressCountDF = addressesRawDF.groupBy("user_id").agg(count("user_id").as("times"))

val userWithCountDF = usersRawDF.join(
addressCountDF,
usersRawDF("id") <=> addressCountDF("user_id")
    && usersRawDF("id") <=> addressCountDF("user_id"),
"left"
)

userWithCountDF.select("id", "times").filter("times > 0").show(100)

Adjust the latest version for Ubuntu
prepare:
    wget http://apache-mirror.8birdsvideo.com/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz -P install/
    wget http://apache.mirrors.ionfish.org/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz -P install/
    wget http://www.gtlib.gatech.edu/pub/apache/zeppelin/zeppelin-0.8.2/zeppelin-0.8.2-bin-all.tgz -P install/

On new ubuntu machine
> sudo apt install make
> sudo snap install docker
> sudo groupadd docker
> sudo gpasswd -a $USER docker
> sudo usermod -aG docker $USER

Here is the Dockerfile
#Run a kafka server side

#Prepare the OS
FROM            ubuntu:16.04
MAINTAINER      Yiyi Kang <[e-mail]>

ENV DEBIAN_FRONTEND noninteractive
ENV JAVA_HOME       /usr/lib/jvm/java-8-openjdk-amd64
ENV LANG            en_US.UTF-8
ENV LC_ALL          en_US.UTF-8

RUN apt-get -qq  update
RUN apt-get -qqy dist-upgrade

#Prepare the denpendencies
RUN apt-get install -qy wget unzip vim
RUN apt-get install -qy iputils-ping

#Install SUN JAVA
RUN apt-get update && \
    apt-get install -y --no-install-recommends locales && \
    locale-gen en_US.UTF-8 && \
    apt-get dist-upgrade -y && \
    apt-get install -qy openjdk-8-jdk

#Prepare for hadoop and spark
RUN apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN ssh-keygen -q -t rsa -N '' -f /root/.ssh/id_rsa
RUN cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

#Prepare python env
RUN apt-get install -y git
RUN apt-get install -y build-essential zlib1g-dev libbz2-dev
RUN apt-get install -y libreadline6 libreadline6-dev sqlite3 libsqlite3-dev
RUN apt-get update --fix-missing
RUN apt-get install -y libssl-dev

RUN apt-get install -y software-properties-common vim
RUN add-apt-repository ppa:jonathonf/python-3.6
RUN apt-get update

RUN apt-get install -y build-essential python3.6 python3.6-dev python3-pip python3.6-venv
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN ln -s /usr/bin/pip3 /usr/bin/pip

#pandas
RUN pip install pandas
RUN pip install -U pandasql

RUN            mkdir /tool/
WORKDIR        /tool/
#add the software hadoop
ADD            install/hadoop-3.2.1.tar.gz /tool/
RUN            ln -s /tool/hadoop-3.2.1 /tool/hadoop
ADD            conf/core-site.xml /tool/hadoop/etc/hadoop/
ADD            conf/hdfs-site.xml /tool/hadoop/etc/hadoop/
ADD            conf/hadoop-env.sh /tool/hadoop/etc/hadoop/

#add the software spark
ADD            install/spark-2.4.4-bin-hadoop2.7.tgz /tool/
RUN            ln -s /tool/spark-2.4.4-bin-hadoop2.7 /tool/spark
ADD            conf/spark-env.sh /tool/spark/conf/

#add the software zeppelin
ADD            install/zeppelin-0.8.2-bin-all.tgz /tool/
RUN            ln -s /tool/zeppelin-0.8.2-bin-all /tool/zeppelin

#set up the app
EXPOSE  9000 9870 8080 4040
RUN     mkdir -p /app/
ADD     start.sh /app/
WORKDIR /app/
CMD    [ "./start.sh" ]

Try to Set Up on Host Machine on CentOS 7
I have JAVA there according to jenv
Need JAVA ENV JDK8, 11, 12
> sudo yum install git
> git clone https://github.com/gcuisinier/jenv.git ~/.jenv
> echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.bash_profile
> echo 'eval "$(jenv init -)"' >> ~/.bash_profile
> . ~/.bash_profile

Check version
> jenv --version
jenv 0.5.2-12-gdcbfd48

Download JDK 8, 11, 12 from Official website
https://www.oracle.com/technetwork/java/javase/downloads/jdk12-downloads-5295953.html
jdk-11.0.4_linux-x64_bin.tar.gz

Unzip all of these files and place in working directory, link to /opt directory
> tar zxvf jdk-11.0.4_linux-x64_bin.tar.gz

> mv jdk-11.0.4 ~/tool/
> sudo ln -s /home/redis/tool/jdk-11.0.4 /opt/jdk-11.0.4

Add to JENV
> jenv add /opt/jdk-11.0.4

Check the installed versions
>  jenv versions
* system (set by /home/redis/.jenv/version)
  11
  11.0
  11.0.4
  oracle64-11.0.4

Try to set global to 11
> jenv global 11.0

> java -version
java version "11.0.4" 2019-07-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.4+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.4+10-LTS, mixed mode)

Prepare HADOOP
> wget http://apache-mirror.8birdsvideo.com/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
> tar zxvf hadoop-3.2.1.tar.gz
> mv hadoop-3.2.1 ~/tool/
> sudo ln -s /home/carl/tool/hadoop-3.2.1 /opt/hadoop-3.2.1
> sudo ln -s /opt/hadoop-3.2.1 /opt/hadoop

Site Configuration
> vi etc/hadoop/core-site.xml
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://0.0.0.0:9000</value>
        </property>
</configuration>

HDFS site configuration
> vi etc/hadoop/hdfs-site.xml
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
</configuration>

Shell Command ENV
Check JAVA_HOME before we configuration the file
> jenv doctor
[OK] No JAVA_HOME set
[OK] Java binaries in path are jenv shims
[OK] Jenv is correctly loaded

> jenv enable-plugin export
Restart the service
> jenv global 11.0
> java -version
java version "11.0.4" 2019-07-16 LTS
> echo $JAVA_HOME
/home/carl/.jenv/versions/11.0


> vi etc/hadoop/hadoop-env.sh
export JAVA_HOME="/home/carl/.jenv/versions/11.0"

SSH to my localhost, promote for password
> ssh localhost

Generate the key pair
> ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa
> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

SSH to localhost successful
> ssh localhost
Last login: Thu Oct 24 16:12:36 2019 from localhost

Start HDFS
> cd /opt/hadoop
> bin/hdfs namenode -format
> sbin/start-dfs.sh

Visit the web UI
http://rancher-worker1:9870/dfshealth.html#tab-overview

Exceptions:
Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error

Find this in the logging
> grep "Error" ./*
./hadoop-carl-namenode-rancher-worker1.log:2019-10-24 16:15:59,844 WARN org.eclipse.jetty.servlet.ServletHandler: Error for /webhdfs/v1/
./hadoop-carl-namenode-rancher-worker1.log:java.lang.NoClassDefFoundError: javax/activation/DataSource
./hadoop-carl-namenode-rancher-worker1.log: at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)

Solution:
https://github.com/highsource/jsonix-schema-compiler/issues/81
https://salmanzg.wordpress.com/2018/02/20/webhdfs-on-hadoop-3-with-java-9/
> vi etc/hadoop/hadoop-env.sh
export HADOOP_OPTS="--add-modules java.activation"
Not working at all, according to this web page
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
Currently it only support JDK8, I will need to change to JDK8 instead.
> jenv global 1.8
> java -version
java version "1.8.0_221"
> echo $JAVA_HOME
/home/carl/.jenv/versions/1.8

Prepare Spark
> wget http://apache.mirrors.ionfish.org/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
> tar zxvf spark-2.4.4-bin-hadoop2.7.tgz
> mv spark-2.4.4-bin-hadoop2.7 ~/tool/spark-2.4.4
> sudo ln -s /home/carl/tool/spark-2.4.4 /opt/spark-2.4.4
> sudo ln -s /opt/spark-2.4.4 /opt/spark
> cd /opt/spark
> cp conf/spark-env.sh.template conf/spark-env.sh

> vi conf/spark-env.sh
HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop

Prepare PYTHON 3.7 ENV
Since we want to migrate all the things to Python3, install and prepare python3
Install PYENV from the latest github
> git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Add to the PATH
> vi ~/.bash_profile
PATH=$PATH:$HOME/.pyenv/bin
eval "$(pyenv init -)"
> . ~/.bash_profile

Check installation
>   pyenv -v
pyenv 1.2.14-8-g0e7cfc3

Check all the versions, latest is 3.7.5 and 3.8.0， install some other versions
https://www.python.org/downloads/

Some warning and possible dependencies
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?

> sudo yum install bzip2-devel
> sudo yum install sqlite-devel
> sudo yum install readline-devel

>  pyenv install 3.8.0
>  pyenv install 3.7.5

>  pyenv versions
* system (set by /home/carl/.pyenv/version)
  3.7.5
  3.8.0

> pyenv global 3.8.0

>  python -V
Python 3.8.0

More Python Libraries
> pip install --upgrade pip
> pip install pandas
> pip install -U pandasql
Failed with
ModuleNotFoundError: No module named '_ctypes'

Solution:
> sudo yum install libffi-devel
It will solve the problem. Success install pandas and pandasql again.
> pip install pandas
> pip install -U pandasql

Prepare Zeppelin
> wget http://www.gtlib.gatech.edu/pub/apache/zeppelin/zeppelin-0.8.2/zeppelin-0.8.2-bin-all.tgz
> tar zxvf zeppelin-0.8.2-bin-all.tgz
> mv zeppelin-0.8.2-bin-all ~/tool/zeppelin-0.8.2
> sudo ln -s /home/carl/tool/zeppelin-0.8.2 /opt/zeppelin-0.8.2
> sudo ln -s /opt/zeppelin-0.8.2 /opt/zeppelin

Some Configuration for Zeppelin
> cp conf/zeppelin-site.xml.template conf/zeppelin-site.xml
> cp conf/shiro.ini.template conf/shiro.ini
> cp conf/zeppelin-env.sh.template conf/zeppelin-env.sh
Add user to the auth config
> vi conf/shiro.ini
[users]
carl = pass123, admin
kiko = pass123, admin

Site configuration
> vi conf/zeppelin-site.xml
<property>
  <name>zeppelin.server.addr</name>
  <value>0.0.0.0</value>
  <description>Server binding address</description>
</property>
<property>
  <name>zeppelin.anonymous.allowed</name>
  <value>false</value>
  <description>Anonymous user allowed by default</description>
</property>

ENV configuration
> vi conf/zeppelin-env.sh
export SPARK_HOME="/opt/spark"
export HADOOP_CONF_DIR="/opt/hadoop/etc/hadoop/"

Start the Service
> bin/zeppelin.sh

> sudo bin/zeppelin-daemon.sh stop

> sudo bin/zeppelin-daemon.sh start

Visit these UI
http://rancher-worker1:9870/explorer.html#/
http://rancher-worker1:8080/#/

References:
Docker Permission
https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo
JAVA_HOME
https://www.jenv.be/
https://github.com/jenv/jenv
https://stackoverflow.com/questions/28615671/set-java-home-to-reflect-jenv-java-version