hadoop(3)Upgrade to YARN and Installation

hadoop(3)Upgrade to YARN and Installation

1. Introduction
Hadoop 1.0
HDFS and MapReduce, HDFS contains NameNode and multiple DataNode, MapReduce contains JobTracker and mutiple TaskTracker.

Hadoop 1.x, 0.21.X, 0.22.x

Hadoop 2.0
YARN(Yet Another Resource Negotiator)
JobTracker ——> ResourceManager, ApplicationMaster
Hadoop 0.23.x, 2.x

1. Installation

Try to build it my self.
Check Protocol Buffer
>protoc --version
libprotoc 2.5.0

Check the Java Version
>java -version
java version "1.6.0_65"

Create binary distribution without native code and without documentation:
>mvn package -Pdist -DskipTests -Dtar

If there is memory issue, then go this command, but till now, there is no memory issue for me
>export MAVEN_OPTS="-Xms256m -Xmx512m"

After build, I get this file in the diet directory. hadoop-2.4.0.tar.gz

Unzip the file
>tar zxvf hadoop-2.4.0.tar.gz
>sudo ln -s /Users/carl/tool/hadoop-2.4.0 /opt/hadoop-2.4.0
>sudo ln -s /opt/hadoop-2.4.0 /opt/hadoop

Edit the environment
export HADOOP_PREFIX=/opt/hadoop
export PATH=/opt/hadoop/bin:$PATH

>hadoop version
Hadoop 2.4.0 Subversion Unknown -r Unknown Compiled by carl on 2014-06-20T21:21Z

2. Standalone Operation
>mkdir input
>cp /opt/hadoop/etc/hadoop/*.xml input/
>hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output 'dfs[a-z.]+'
>cat output/*

That is NON distribute mode. Single Java Process.

3. Pseudo Distributed Mode
Each Hadoop Daemon runs in a separate Java Process.

Setup ssh
I can directly ssh on my localhost. So I thought I already have done this.
>ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
>cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Configuration
>vi etc/hadoop/core-site.xml

<configuration> 

  <property>       

    <name>fs.defaultFS</name>       

    <value>hdfs://localhost:9000</value> 

  </property>

</configuration>

>vi etc/hadoop/hdfs-site.xml

<configuration> 

  <property>       

    <name>dfs.replication</name>       

    <value>1</value> 

  </property>

</configuration>

Run the MapReduce job locally.

Format the filesystem
>hdfs namenode -format

Start the HDFS
>sbin/start-dfs.sh

Visit the overview pages from WEB UI
http://localhost:50070/dfshealth.html#tab-overview

Create the directories on HDFS
>hdfs dfs -mkdir /user
>hdfs dfs -mkdir /user/sillycat

Warning Message
2014-06-23 11:05:44.531 java[10276:1003] Unable to load realm info from SCDynamicStore 14/06/23 11:05:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I know that I did not compile that native-hadoop library, so never mind.

Put the my local files to HDFS
>hdfs dfs -put etc/hadoop /user/sillycat/input

Run the hadoop command to start the jobs on HDFS
>hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep /user/sillycat/input /user/sillycat/output 'dfs[a-z.]+'

Fetch the files from HDFS to local
>hdfs dfs -get /user/sillycat/output /Users/carl/work/hadoop/output2

Check the results from local disk
>cat /Users/carl/work/hadoop/output2/*

Chck the results from HDFS
>hdfs dfs -cat /user/sillycat/output/*

Stop the HDFS
>sbin/stop-dfs.sh

Run MapReduce Job on YARN
Prepare the Configuration
>cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml

>vi etc/hadoop/mapred-site.xml

<configuration> 

  <property>       

    <name>mapreduce.framework.name</name>       

    <value>yarn</value> 

  </property>

</configuration>


>vi etc/hadoop/yarn-site.xml

<configuration> 

  <property>       

   <name>yarn.nodemanager.aux-services</name>              

   <value>mapreduce_shuffle</value> 

  </property>

</configuration>

Start the HDFS
>sbin/start-dfs.sh

Start the YARN
>sbin/start-yarn.sh

Web UI to track the jobs
http://localhost:8088/cluster

Setup the configuration files
>hdfs dfs -put /opt/hadoop/etc/hadoop /user/carl/input

Run the tasks
>hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output 'dfs[a-z.]+'

But after I run the job, I did not see I get a good results. Then I go and check the log file from here.
/HADOOP_HOME/logs/yarn-carl-nodemanager-sparkworker1.local.log

Error Message
java.lang.RuntimeException: No class defined for mapreduce_shffle        at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:109)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

Solution:
I get a typo in the configuration file. Change that. This should be the right value <value>mapreduce_shuffle</value>

Error Message:
14/06/23 12:20:21 INFO mapreduce.Job: Job job_1403543828091_0003 failed with state FAILED due to: Application application_1403543828091_0003 failed 2 times due to AM Container for appattempt_1403543828091_0003_000002 exited with  exitCode: 127 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)

Solution:
Find the log file from here http://localhost:8042/node/containerlogs/container_1403543828091_0003_01_000001/carl/stderr/?start=-4096

I saw this kind of error
/bin/bash: /bin/java: No such file or directory

>sudo ln -s /usr/bin/java /bin/java

Then run the command to execute the job
>hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output 'dfs[a-z.]+'

It works.

References:
http://sillycat.iteye.com/blog/1556106
http://sillycat.iteye.com/blog/1556107

http://www.iteblog.com/archives/category/hadoop
http://my.oschina.net/leejun2005/blog/97802

Official Document
http://hadoop.apache.org/docs/r2.4.0/
http://www.bizdirusa.com/mirrors/apache/hadoop/common/ download URL