Spark(7)Spark 1.0.2 on Mesos

Spark(7)Spark 1.0.2 on Mesos

Spark 1.0.2 is designed for use with Mesos 0.18.1

1. Install and Set up Mesos
Haha, I used to have problem based run spark on Mesos. Good luck this time.
>wget http://archive.apache.org/dist/mesos/0.18.1/mesos-0.18.1.tar.gz

Install the compile library
>sudo apt-get install build-essential
>sudo apt-get install python-dev
>sudo apt-get install libz-dev
>sudo apt-get install libcurl3 libcurl3-gnutls libcurl4-openssl-dev
>sudo apt-get install libsasl2-dev

Configure the generate the make file
>./configure --with-python-headers=/usr/include/python2.7 --with-java-home=/usr/lib/jvm/java-7-oracle --with-java-headers=/usr/lib/jvm/java-7-oracle/include --with-webui --with-included-zookeeper --prefix=/home/carl/tool/mesos-0.18.1

>make

>make install

Link and place everything in PATH.

Check and Configure the Mesos configuration
>cp var/mesos/deploy/mesos-deploy-env.sh.template var/mesos/deploy/mesos-deploy-env.sh
>cp var/mesos/deploy/mesos-master-env.sh.template var/mesos/deploy/mesos-master-env.sh
Open the log configuration
>sudo mkdir /var/log/mesos
>sudo chmod 777 /var/log/mesos/

>cp var/mesos/deploy/mesos-slave-env.sh.template var/mesos/deploy/mesos-slave-env.sh
Set the configurations
export MESOS_master=ubuntu-master1:5050# Other options you're likely to want to set:export MESOS_log_dir=/var/log/mesosexport MESOS_work_dir=/var/run/mesosexport MESOS_isolation=cgroups

>sudo mkdir /var/run/mesos
>sudo chmod 777 /var/run/mesos

setting masters
>sudo vi var/mesos/deploy/masters
ubuntu-master1

setting slaves
>sudo vi var/mesos/deploy/slaves
ubuntu-slave1
ubuntu-slave2
ubuntu-slave3

Set the root ssh key access to ubuntu-master1, ubuntu-slave1, ubuntu-slave2, ubuntu-slave3. And then run the cluster of Mesos
>sudo sbin/mesos-start-cluster.sh

Visit the cluster UI
http://ubuntu-master1:5050/#/

2. Set up and Run Spark
Add the Mesos library to the conf
>vi conf/spark-env.sh
export MESOS_NATIVE_LIBRARY=/opt/mesos/lib/libmesos.so

Upload the Spark Package
place the file in http or s3 or hdfs.

Start HDFS
>sbin/start-dfs.sh

Download the binary file
>wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2-bin-hadoop2.tgz
>hdfs dfs -mkdir /user
>hdfs dfs -mkdir /user/sillycat
>hdfs dfs -put ./spark-1.0.2-bin-hadoop2.tgz /user/sillycat/


The framework runs great on Mesos.

>bin/spark-submit --class com.sillycat.spark.app.FindWordJob --master mesos://ubuntu-master1:5050 /home/carl/work/sillycat-spark/target/scala-2.10/sillycat-spark-assembly-1.0.jar book1 hdfs://ubuntu-master1:9000/user/sillycat/spark-1.0.2-bin-hadoop2.tgz


References:
http://spark.apache.org/docs/latest/running-on-mesos.html
http://sillycat.iteye.com/blog/2083194
http://mesos.apache.org/gettingstarted/

http://dongxicheng.org/framework-on-yarn/apache-spark-comparing-three-deploying-ways/