Hadoop “Unable to load native-hadoop library for your platform” warning

Hadoop “Unable to load native-hadoop library for your platform” warning

 

Hadoop Installation on Mac OS X：

https://isaacchanghau.github.io/post/install_hadoop_mac/

 

Compile Apache Hadoop on Linux (fix warning: Unable to load native-hadoop library)

http://www.ercoppa.org/posts/how-to-compile-apache-hadoop-on-ubuntu-linux.html

 

 

Question:

I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I'm running Hadoop 2.2.0.

 

Answer:

I assume you're running Hadoop on 64bit CentOS. The reason you saw that warning is the native Hadoop library $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 was actually compiled on 32 bit.

Anyway, it's just a warning, and won't impact Hadoop's functionalities.

Here is the way if you do want to eliminate this warning, download the source code of Hadoop and recompile libhadoop.so.1.0.0 on 64bit system, then replace the 32bit one.

Steps on how to recompile source code are included here for Ubuntu:

http://www.ercoppa.org/Linux-Compile-Hadoop-220-fix-Unable-to-load-native-hadoop-library.htm

Good luck.

 

 

 

 

 

 

 

 

 

 

 

 

Compile Apache Hadoop on Linux (fix warning: Unable to load native-hadoop library)

This tutorial explains how to compile Apache Hadoop 2.4.0 under Ubuntu Linux 13.10 amd64. You may want to compile Hadoop in order to fix the issue:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

• Install a JDK: Oracle JDK (suggested) or package openjdk-7-jdk
• Install maven, libssl-dev, build-essential, pkgconf, and cmake.

• Install the library protobuf:

  • If you are running Ubuntu 13.10 or earlier then install locally protobuf-2.5.0 and insert it in the PATH:

    cd protobuf-2.5.0/
    ./configure --prefix=`pwd`/inst/bin && make && make install
    export PATH=`pwd`/inst/bin:$PATH

  • Otherwise (Ubuntu 14.04 or newer):

    sudo apt-get install libprotobuf8 protobuf-compiler

• Download Hadoop sources.

• Compile Apache Hadoop:

  tar xvf hadoop-2.4.0-src.tar.gz
  cd hadoop-2.4.0-src
  mvn package -Pdist,native -DskipTests -Dtar

  Check hadoop-dist/target/hadoop-2.4.0.tar.gz (e.g., use this as your hadoop binary) or hadoop-dist/target/hadoop-2.4.0. If you have already installed a 32bit Hadoop, then you need only to replace the native libs in $HADOOP/lib/ with the new native libs (hadoop-dist/target/hadoop-2.4.0/lib) and remove (if applicable) from $HADOOP/etc/hadoop-env.sh:

  export HADOOP_COMMON_LIB_NATIVE_DIR="~/hadoop/lib/"
  export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=~/hadoop/lib/"

• You can delete Hadoop (and Protobuf sources if Ubuntu 13.10 or earlier).