hadoop(6)2.6.0 and Example

hadoop(6)2.6.0 and Example

 

1. Prepare

ProtocolBuffer Installation

> git clone https://github.com/google/protobuf.git

> ./autogen.sh 

> ./configure --prefix=/home/carl/tool/protobuf-3.0.0

> make

> make check

> make install

 

Add that to path.

> protoc --version

libprotoc 3.0.0

 

Need to switch to 2.5.0 version

https://code.google.com/p/protobuf/downloads/list

 

JDK Installation

> java -version
java version "1.6.0_45"

Java(TM) SE Runtime Environment (build 1.6.0_

 

MAVEN Installation

> wget http://mirrors.gigenet.com/apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz

 

Unzip and place in the right directory, adding to path

> mvn --version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)

Maven home: /opt/maven

 

Add this to stop the out of memory error

export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=128m"

 

Hadoop Src Build

> wget http://apache.claz.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0-src.tar.gz

> mvn package -Pdist -DskipTests -Dtar

 

Find the release dist file  hadoop-2.6.0.tar.gz

 

I get “JAVA_HOME not set” Error message, I already have the JAVA_HOME in .profile, but I added it in 

/opt/hadoop/etc/hadoop/hadoop-env.sh as well.

export JAVA_HOME="/usr/lib/jvm/java-8-oracle"

 

Configure the SSH on master

> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

 

Format file system

Format the filesystem

>hdfs namenode -format

 

All the configuration should take reference from 

http://sillycat.iteye.com/blog/2090186

 

It works. Change the etc/hadoop/slaves files to have a machine list there.

 

Command line to start the HDFS and YARN

> sbin/start-dfs.sh

> sbin/start-yarn.sh

 

 

Tips

Error Message:

autoreconf: not found

Solution:

>sudo apt-get install autoconf

 

Error Message:

Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN7> line 6.
autoreconf: failed to run libtoolize: No such file or directory

autoreconf: libtoolize is needed because this package uses Libtool

Solution:

> sudo apt-get install libtool

 

Error Message:

checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/carl/install/protobuf':

configure: error: C++ preprocessor "/lib/cpp" fails sanity check

Solution:

> sudo apt-get install g++

 

Error Message:

protoc version is 'libprotoc 3.0.0', expected version is '2.5.0'

Solution:

Switch to 2.5.0

 

References:

http://sillycat.iteye.com/blog/1556106  Hadoop doc and Installation on ubuntu with an old version

http://sillycat.iteye.com/blog/1556107 cluster of the old version

http://sillycat.iteye.com/blog/2084169  yet another resource negotiator 2.4.0

http://sillycat.iteye.com/blog/2090186 cluster 2.4.1

http://sillycat.iteye.com/blog/2090937 eclipse and example