hadoop使用LZO压缩

1。安装LZO
sudo apt-get install liblzo2-dev
或者下载lzo2
http://www.oberhumer.com/opensource/lzo/download/.
./configure --enable-shared
make
make install


确认安装成功

% ls -l /usr/lib*/liblzo2*
-rw-r--r--  1 root root 171056 Mar 20  2006 /usr/lib/liblzo2.a
lrwxrwxrwx  1 root root     16 Feb 17  2007 /usr/lib/liblzo2.so -> liblzo2.so.2.0.0*
lrwxrwxrwx  1 root root     16 Feb 17  2007 /usr/lib/liblzo2.so.2 -> liblzo2.so.2.0.0*
-rwxr-xr-x  1 root root 129067 Mar 20  2006 /usr/lib/liblzo2.so.2.0.0*
-rw-r--r--  1 root root 208494 Mar 20  2006 /usr/lib64/liblzo2.a
lrwxrwxrwx  1 root root     16 Feb 17  2007 /usr/lib64/liblzo2.so -> liblzo2.so.2.0.0*
lrwxrwxrwx  1 root root     16 Feb 17  2007 /usr/lib64/liblzo2.so.2 -> liblzo2.so.2.0.0*
-rwxr-xr-x  1 root root 126572 Mar 20  2006 /usr/lib64/liblzo2.so.2.0.0*

2。下载hadoop-gpl-compression源码

http://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/downloads/list

3.编译

export JAVA_HOME=/path/to/64bit/jdk
export CFLAGS=-m64
export CXXFLAGS=-m64
ant compile-native tar

4.拷贝生成的jar到指定目录

cp build/hadoop-gpl-compression-0.1.0-dev/hadoop-gpl-compression-0.1.0-dev.jar /path/to/hadoop/dist/lib

5.拷贝native library

tar -cBf - -C build/hadoop-gpl-compression-0.1.0-dev/lib/native . | tar -xBvf - -C /path/to/hadoop/dist/lib/native

6.修改hadoop配置文件core-site.xml

<property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
  </property>
  <property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
  </property>

7.修改hadoop配置文件mapred-site.xml

<property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
  </property>
  <property>
    <name>mapred.map.output.compression.codec</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
  </property>

8下载一个windows的lzop的客户端，生存一个test.lzo

9.生存索引
单机版

hadoop jar /path/to/your/hadoop-lzo.jar com.hadoop.compression.lzo.LzoIndexer big_file.lzo

集群版

hadoop jar /path/to/your/hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer big_file.lzo

10.执行一个mapred程序，如wordcount
添加参数"-inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat"

评论

3 楼 heipark 2011-10-09  

最新版的hadoop lzo库在这里：https://github.com/kevinweil/hadoop-lzo，google code那个不被推荐了。

2 楼 奔跑的羚羊 2011-10-08  

下载源码，编译以后生成的jar

1 楼 sawen21 2011-09-27  

请问你这里的hadoop-lzo.jar是哪来的？