hadoop2.x的编译 & hadoop-snappy编译 & hadoop2.x 配置 snappy压缩

hadoop2.x 默认没有支持snappy压缩，需要我们自己编译 才能支持snappy的压缩。

查看hadoop2.x 的BUILDING文件 ，

 Snappy build options:

   Snappy is a compression library that can be utilized by the native code.

   It is currently an optional component, meaning that Hadoop can be built with

   or without this dependency.

 

  * Use -Drequire.snappy to fail the build if libsnappy.so is not found.

    If this option is not specified and the snappy library is missing,

    we silently build a version of libhadoop.so that cannot make use of snappy.

    This option is recommended if you plan on making use of snappy and want

    to get more repeatable builds.

 

 1.环境准备

• 安装 jdk1.6
• 安装maven-3.05
• 安装gcc,openssl,cmake 等软件

 

sudo yum -y install gcc-c++ libstdc++-devel
sudo yum -y install openssl openssl-devel autoconf ncurses-devel libtool cmake zlib-devel

 

 

• 安装protobuf-2.5.0

   下载 http://pan.baidu.com/s/1pJlZubT

 

tar -zxvf protobuf-2.5.0.tar.gz
cd protobuf
sudo ./configure
sudo make & make install

#验证：
protoc --version

 

 

 

• 安装findbugs

 

wget -c http://prdownloads.sourceforge.net/findbugs/findbugs-2.0.3.tar.gz?download
tar -zxvf findbugs-2.0.3.tar.gz -C /opt/modules

#配置FINDBUGS环境变量
sudo vi /etc/profile

 

export FINDBUGS_HOME=/opt/findbugs-2.0.3
export PATH=$PATH:$FINDBUGS_HOME/bin

source /etc/profile

 

 

#验证：

 fb -version

 

 

 

• 安装snappy压缩库

       下载snappy-1.1.1.tar.gz http://pan.baidu.com/s/1o6vGQ18

 

tar -zxvf snappy-1.1.1.tar.gz
cd snappy-1.1.1 
sudo ./configure
sudo make
sudo make install

 

 

 2.编译hadoop 2.x，对应系统的本地库，其中要包含支持snappy lib

    下载hadoop2.x   我选用hadoop-2.5.0-cdh5.3.3

 

wget -c http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.5.0-cdh5.3.3-src.tar.gz
tar -zxvf hadoop-2.5.0-cdh5.3.3-src.tar.gz -C /opt/compile
cd /opt/compile/hadoop-2.5.0-cdh5.3.3
#必须加上 -Drequire.snappy 参数才能支持snappy压缩
mvn clean package -Pdist,native -Drequire.snappy -DskipTests -Dtar

#编译的native文件 http://pan.baidu.com/s/12QVam

 

3.替换hadoop lib/native包

把编译之后的lib/native文件替换掉$HADOOP_HOME/lib/native文件

注意事项：

替换前后必须保持所有文件的格式一致（所有的.a文件 权限655 ，所有的 *.so.xx.xx 权限 755 ,给*.so.xx.xx 创建软链接）

 

cd $HADOOP_HOME/lib/
mv native backup_native
mkdir native
cp 自己编译的/lib/native/* native/
chmod 644 *.a
rm libhadoop.so libhdfs.so libnativetask.so
chmod 755 libnativetask.so.1.0.0
chmod 755 libhdfs.so.0.0.0
chmod 755 libhadoop.so.1.0.0
ln -s libnativetask.so.1.0.0 libnativetask.so
ln -s libhdfs.so.0.0.0 libhdfs.so
ln -s libhadoop.so.1.0.0 libhadoop.so
#ll  各个文件格式对应如下:

-rw-r--r-- 1 ehp ehp 1178622 Jul  5 08:37 libhadoop.a
-rw-r--r-- 1 ehp ehp 1487052 Jul  5 08:37 libhadooppipes.a
lrwxrwxrwx 1 ehp ehp      18 Jul  5 08:38 libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 1 ehp ehp  697211 Jul  5 08:37 libhadoop.so.1.0.0
-rw-r--r-- 1 ehp ehp  582056 Jul  5 08:37 libhadooputils.a
-rw-r--r-- 1 ehp ehp  359794 Jul  5 08:37 libhdfs.a
lrwxrwxrwx 1 ehp ehp      16 Jul  5 08:38 libhdfs.so -> libhdfs.so.0.0.0
-rwxr-xr-x 1 ehp ehp  228715 Jul  5 08:37 libhdfs.so.0.0.0
-rw-r--r-- 1 ehp ehp 7684428 Jul  5 08:37 libnativetask.a
lrwxrwxrwx 1 ehp ehp      22 Jul  5 08:39 libnativetask.so -> libnativetask.so.1.0.0
-rwxr-xr-x 1 ehp ehp 3061007 Jul  5 08:37 libnativetask.so.1.0.0

4.编译hadoop-snappy库

下载地址：http://code.google.com/p/hadoop-snappy/(可以无法访问)

我的云盘分享地址 http://pan.baidu.com/s/1pJkj7sB

 

Build Hadoop Snappy

1. Requirements: gcc c++, autoconf, automake, libtool, Java 6, JAVA_HOME set, Maven 3

2. Build/install Snappy (http://code.google.com/p/snappy/)

3. Build Hadoop Snappy

$ mvn package [-Dsnappy.prefix=SNAPPY_INSTALLATION_DIR]

'snappy.prefix' by default is '/usr/local'. If Snappy is installed in other location than user local set 'snappy.prefix' to the right location.

The built tarball is at target/hadoop-snappy-0.0.1-SNAPSHOT.tar.gz. The tarball includes snappy native library

 

编译后文件：下载地址 http://pan.baidu.com/s/1hq5w7cg

 

放入hadoop2到对应的目录下

$ tar -zxf hadoop-snappy-0.0.1-SNAPSHOT.tar.gz
$ cd hadoop-snappy-0.0.1-SNAPSHOT
$ cp -r lib/native/Linux-amd64-64/* /opt/modules/hadoop-2.5.0-cdh5.3.3/lib/native
$ cp lib/hadoop-snappy-0.0.1-SNAPSHOT.jar /opt/modules/hadoop-2.5.0-cdh5.3.3/share/hadoop/common/lib/

5.设置mapper output 启动压缩和压缩编码

 

#mapred-site.xml
<property>
	<name>mapreduce.map.output.compress</name>
	<value>true</value>
</property>
<property>
	<name>mapreduce.map.output.compress.codec</name>
	<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

 

6.检查snappy是否正确配置

hadoop checknative

7.测试

hadoop fs -rm -R /user/ehp/mapred/wordcount/output

bin/yarn jar \

share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.3.jar \

wordcount \

/user/ehp/mapred/wordcount/input \

/user/ehp/mapred/wordcount/output

 

在uber模式下我们可以看到这么一个问题

 

2015-03-18 21:28:20,153 FATAL [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Error running local (uberized) 'child' : java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
        at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)

 

解决办法：https://issues.apache.org/jira/browse/MAPREDUCE-5799

#yarn-site.xml
<property>
 <name>yarn.app.mapreduce.am.env</name>
 <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native</value>
 </property>

 

不修改yarn-site.xml 运行时动态增加配置：

bin/yarn jar \
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.3.jar \
wordcount \
-Dyarn.app.mapreduce.am.env=LD_LIBRARY_PATH=/opt/modules/hadoop-2.5.0-cdh5.3.3/lib/native \
/user/ehp/mapred/wordcount/input \
/user/ehp/mapred/wordcount/output