DevOps(7)Spark Deployment on VM with Kafka

DevOps(7)Spark Deployment on VM with Kafka

 

1. Install the zookeeper

I pick up this version http://apache.mirrors.lucidnetworks.net/zookeeper/stable/zookeeper-3.4.6.tar.gz.

 

> sudo ln -s /home/carl/tool/zookeeper-3.4.6 /opt/zookeeper-3.4.6

> sudo ln -s /opt/zookeeper-3.4.6 /opt/zookeeper

 

Add it to the path

> cp conf/zoo_sample.cfg conf/zoo.cfg

 

Start the server

> zkServer.sh start zoo.cfg

 

Connect to the server

> zkCli.sh -server localhost:2181

 

Stop the server

> zkServer.sh stop

 

2. Install kafka

I pick up this version http://www.carfab.com/apachesoftware/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz

 

Build the source

> ./gradlew -PscalaVersion=2.10.0 releaseTarGz -x signArchives

 

Find the binary file here

> cd core/build/distributions/

 

Place the binary file to working directory and add it to the path.

 

Adjust the config file  vi config/server.properties 

 

Command to start kafka

nohup bin/kafka-server-start.sh config/server-dev2.properties &

 

Install the kafka monitor

> git clone https://github.com/quantifind/KafkaOffsetMonitor.git

 

Build and generate the assembly jar 

> sbt clean update

> sbt assembly

 

Similar command to start that

> java -cp /opt/kafka-monitor/lib/*.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk ubuntu-dev1,ubuntu-dev2 --port8082--refresh 10.seconds --retain 2.days

 

Tip:

Some Mysql Timezone Config

> mysql_tzinfo_to_sql /usr/share/zoneinfo/ | mysql -u root -p mysql

 

Or

 

mysql_tzinfo_to_sql /usr/share/zoneinfo/ > mysql.sql

 

Fix the too long string manually

> cat mysql.sql | mysql -u root -p mysql

 

3. Install rabbitMQ

http://sillycat.iteye.com/admin/blogs/2183555

http://sillycat.iteye.com/blog/1565771

 

Install erlang from source

>sudo apt-get install build-essential libncurses5-dev openssl libssl-dev fop xsltproc unixodbc-dev

 

http://www.erlang.org/download/otp_src_17.4.tar.gz

 

Install rabbitmq from source

http://www.rabbitmq.com/releases/rabbitmq-server/v3.4.3/rabbitmq-server-generic-unix-3.4.3.tar.gz

 

Command to start rabbitmq

>sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit1 sbin/rabbitmq-server -detached

 

When I use guest/guest to login, I get this error message:

=ERROR REPORT==== 5-Feb-2015::13:14:08 ===
webmachine error: path="/api/whoami"

"Unauthorized"

 

Solution:

That is a new feature, I need to solve it like this

http://stackoverflow.com/questions/22850546/cant-access-rabbitmq-web-management-interface-after-fresh-install

 

Creating User, set group, set permission

> rabbitmqctl -n rabbit1 add_user sillycat xxxxx

> rabbitmqctl -n rabbit1 set_user_tags sillycat administrator

> rabbitmqctl -n rabbit1 set_permissions -p / sillycat ".*" ".*" ".*"

 

Start the App and Visit the console

cd /opt/rabbitmq
sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit2 sbin/rabbitmq-server -detached

enable cluster

sudo sbin/rabbitmqctl -n rabbit2 stop_app
sudo sbin/rabbitmqctl -n rabbit2 join_cluster rabbit1@ubuntu-dev1
sudo sbin/rabbitmqctl -n rabbit2 start_app

visit page

http://ubuntu-dev2:15672

 

References:

http://sillycat.iteye.com/blog/2166583

http://sillycat.iteye.com/blog/2167216

 

http://sillycat.iteye.com/blog/2015175

 

http://sillycat.iteye.com/blog/2066116