编译 Hadoop-append 用于Hbase

Hbase基于 hadoop，若 Hbase 直接使用 release 版本的 hadoop，可能出现数据丢失的情况，hbase 需要使用hadoop-append ，详细介绍参考Hbase 官网资料

下面以hbase-0.90.2 为例，介绍hadoop-0.20.2-append 的编译，以下操作参考：

Building an Hadoop 0.20.x version for HBase 0.90.2

操作系统：CentOS Linux

所需工具：git、ant

[zhankunlin@IctHTC ~]$ mkdir hadoop-append-for-hbase
[zhankunlin@IctHTC ~]$ cd hadoop-append-for-hbase
[zhankunlin@IctHTC hadoop-append-for-hbase]$ git clone http://git.apache.org/hadoop-common.git
[zhankunlin@IctHTC hadoop-append-for-hbase]$ cd hadoop-common
[zhankunlin@IctHTC hadoop-common]$ git checkout -t origin/branch-0.20-append
[zhankunlin@IctHTC hadoop-common]$ git show-branch branch-0.20-append
[zhankunlin@IctHTC hadoop-common]$ git show-branch release-0.20.2
[zhankunlin@IctHTC hadoop-common]$ vi ../build.properties
#this is essential
resolvers=internal
#you can increment this number as you see fit
version=0.20-append-for-hbase-zkl
project.version=${version}
hadoop.version=${version}
hadoop-core.version=${version}
hadoop-hdfs.version=${version}
hadoop-mapred.version=${version}
[zhankunlin@IctHTC hadoop-common]$ ln -s ../build.properties build.properties
[zhankunlin@IctHTC hadoop-common]$ pwd
/home/zhankunlin/hadoop-append-for-hbase/hadoop-common

安装 ant
[zhankunlin@IctHTC ~]$ tar zxf apache-ant-1.8.2-bin.tar.gz

[zhankunlin@IctHTC ~]$ export PATH=/home/zhankunlin/apache-ant-1.8.2/bin:$PATH

编译 hadoop

[zhankunlin@IctHTC ~]$ cd hadoop-append-for-hbase/hadoop-common
[zhankunlin@IctHTC hadoop-common]$ git checkout branch-0.20-append确保当前在使用 branch-0.20-append
Already on 'branch-0.20-append'
[zhankunlin@IctHTC hadoop-common]$ ant mvn-install
Buildfile: /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build.xml
.
.
.
mvn-install:
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-core-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-core/0.20-append-for-hbase-zkl/hadoop-core-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-test-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-test/0.20-append-for-hbase-zkl/hadoop-test-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-tools-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-tools/0.20-append-for-hbase-zkl/hadoop-tools-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-examples-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-examples/0.20-append-for-hbase-zkl/hadoop-examples-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/contrib/streaming/hadoop-streaming-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-streaming/0.20-append-for-hbase-zkl/hadoop-streaming-0.20-append-for-hbase-zkl.jar

BUILD SUCCESSFUL
Total time: 11 minutes 42 seconds

可选操作：
[zhankunlin@IctHTC hadoop-common]$ ant test测试所有功能，花很长时间
[zhankunlin@IctHTC hadoop-common]$ ant test-core 测试核心功能

查看编译好的 jar包(在当前用户主目录下)
[zhankunlin@IctHTC hadoop-common]$ find ~/.m2/repository -name "hadoop-*.jar"
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-tools/0.20-append-for-hbase-zkl/hadoop-tools-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-test/0.20-append-for-hbase-zkl/hadoop-test-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-examples/0.20-append-for-hbase-zkl/hadoop-examples-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-streaming/0.20-append-for-hbase-zkl/hadoop-streaming-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-core/0.20-append-for-hbase-zkl/hadoop-core-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-core/0.20.2/hadoop-core-0.20.2.jar

编译好的包有：
[zhankunlin@IctHTC hadoop-common]$ find ~/.m2/repository -name "hadoop-*.jar" | awk -F"/" '{print $11}'
hadoop-tools-0.20-append-for-hbase-zkl.jar
hadoop-test-0.20-append-for-hbase-zkl.jar
hadoop-examples-0.20-append-for-hbase-zkl.jar
hadoop-streaming-0.20-append-for-hbase-zkl.jar
hadoop-core-0.20-append-for-hbase-zkl.jar
hadoop-core-0.20.2.jar (这个包不需要)

注意：
这里 0.20-append-for-hbase-zkl 是在build.properties文件中定义的版本好号 version 值,即：

hadoop-tools-VERSION.jar
hadoop-test-VERSION.jar
hadoop-examples-VERSION.jar
hadoop-streaming-VERSION.jar
hadoop-core-VERSION.jar

将编译好的 jar 包复制出来.
[zhankunlin@IctHTC hadoop-common]$ mk ../ok_jars
[zhankunlin@IctHTC hadoop-common]$ jars=`find ~/.m2/repository -name "hadoop-*.jar"`; for jar in $jars; do cp $jar ../ok_jars/ ; done

接下来就是将编译生成的新jar替换 hadoop-0.20.2-release 和 hbase-0.90.2 中旧的jar包

(1) 替换旧的hadoop包；

请注意，替换 hadoop 中旧的jar包前需要对编译生成的新jar包重新命名。

Hadoop 0.20.2 release 版本中jar包的命名规则为 hadoop-VERSION-PACKAGE.jar,如：hadoop-0.20.2-examples.jar。

而新编译的jar包的命名规则为 hadoop-PACKAGE-VERSION.jar,如： hadoop-examples-0.20-append-for-hbase.jar.

所以需要将新编译的jar包按如下方式重命名以符合hadoop：

hadoop-examples-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-examples.jar
hadoop-test-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-test.jar
hadoop-tools-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-tools.jar
hadoop-streaming-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-streaming.jar
hadoop-core-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-core.jar

(2) 替换hbase中lib文件夹中的包

Hbase使用的命名规则为hadoop-PACKAGE-VERSION.jar，和新编译的jar包命名规则一致，所以只需要直接复制jar包到$HBASE_HOME/lib目录下，不需要重命名。