Install hadoop-lzo-lib for Hadoop v1.1.1 on Mac OS X with Mountain Lion

Here is a quick guide to install hadoop lzo library, the install process is pretty easy actually.

 

• Install LZO with HomeBrew, LZO is a wonderful compression scheme to use with Hadoop. HomeBrew is prerequisite, visit https://github.com/mxcl/homebrew to see how to install HomeBrew.

brew install lzo

  

• Clone the hadoop-lzo source code to local:

git clone https://github.com/twitter/hadoop-lzo.git ~/code/hadoop-lzo && cd ~/code/hadoop-lzo

 

• Build hadoop-lzo to generate hadoop-lzo-lib as well as hadoop lzo native library, find the built artifacts in the target directory.

C_INCLUDE_PATH=/usr/local/Cellar/lzo/2.06/include \
LIBRARY_PATH=/usr/local/Cellar/lzo/2.06/lib \
mvn clean test package

 Waiting for build complete. If something wrong, refer to https://github.com/twitter/hadoop-lzo for troubleshoting.

• Copy the built artifacts into hadoop lib directory or native library directory respectively.

 

• Once the libs are built and installed, you may want to add them to the class paths and library paths. That is, in hadoop-env.sh, set 

export HADOOP_CLASSPATH=$HADOOP_HOME/lib/hadoop-lzo-xxxx.jar
export JAVA_LIBRARY_PATH=/path/to/hadoop-lzo-native-libs:/path/to/standard-hadoop-native-libs

 

• Note that there seems to be a bug in /path/to/hadoop/bin/hadoop; (refer to https://issues.apache.org/jira/browse/HADOOP-6453) . Comment out the line

    JAVA_LIBRARY_PATH=''

 

See https://github.com/twitter/hadoop-lzo for further instruction if you have any question or leave a comment.