Ubuntu 12.04 上使用Hadoop 2.2.0 三 编译64bit版本

下载源代码：

wget http://mirror.esocc.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz

然后解压：

tar zxvf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src

运行下面的命令开始编译：

~/code/hadoop-2.2.0-src$ mvn package -Pdist,native -DskipTests -Dtar

下载了很多maven的东东后，编译报错：

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle
[ERROR]   class file for org.mortbay.component.AbstractLifeCycle not found
/home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle
[ERROR]   class file for org.mortbay.component.LifeCycle not found

编辑hadoop-common-project/hadoop-auth/pom.xml文件，添加依赖：

    <dependency>
      <groupId>org.mortbay.jetty</groupId>
      <artifactId>jetty-util</artifactId>
      <scope>test</scope>
    </dependency>

再次编译，这个错误解决了。

之后遇到了没有安装protocol buffer库的错误，安装一下：

[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common ---
[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, No such file or directory

sudo apt-get install libprotobuf-dev

然后再次运行maven命令编译。发现新的错误，原来ubuntu的安装包里面没有带上protoc编译器。算了，下源代码自己编译。

hduser@hm1:/usr/src$ sudo wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
... 
$ sudo ./configure
$ sudo make
$ sudo make check
$ sudo make install
$ sudo ldconfig
$ protoc --version

hduser@hm1:~$ start-dfs.sh 
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.

再来编译。等...., 说明hadoop不准备64bit的库浪费了大家多少时间。

还有错：

Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
     [exec]   system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES

安装openssl库

sudo apt-get install libssl-dev

再来一次，编译成功了。在目录/home/hduser/code/hadoop-2.2.0-src/hadoop-dist/target 下有文件：

hadoop-2.2.0.tar.gz

解压后，进入目录，然后复制native 目录里的东西到制定位置，覆盖32bit文件

sudo cp -r ./hadoop-2.2.0/lib/native/* /usr/local/hadoop/lib/native/

现在回到~目录，运行下面的命令，成功了。

hduser@hm1:~$ start-dfs.sh 
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.

hduser@hm1:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hm1.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hm1.out

hduser@hm1:~$ jps
32417 Jps
1101 DataNode
1346 SecondaryNameNode
983 NameNode

现在运行例子程序， 注意/etc/hosts中我配置了

127.0.0.1 hm1

hduser@hm1:~/code/hadoop-2.2.0-src$ cd /usr/local/hadoop/
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps  = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:07:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/21 18:07:30 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumC\
fountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:31 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:32 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

需要在yarn-site.xml中添加写配置：

  <property>
    <name>yarn.resourcemanager.address</name>
    <value>127.0.0.1:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>127.0.0.1:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>127.0.0.1:8031</value>
  </property>

重新启动虚拟机，再次启动服务，现在连接问题解决了。

hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5  
Number of Maps  = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:15:53 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:54 INFO input.FileInputFormat: Total input paths to process : 2
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: number of splits:2
14/02/21 18:15:54 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.specul\
ative
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393006528872_0001
14/02/21 18:15:54 INFO impl.YarnClientImpl: Submitted application application_1393006528872_0001 to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:55 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1393006528872_0001/
14/02/21 18:15:55 INFO mapreduce.Job: Running job: job_1393006528872_0001

如果重来一次的话，顺序应该是先编译64bit版本，然后用这个版本进行配置安装。