ubuntu14安装hadoop2

ubuntu

启用root用户： sudo passwd root

查看版本：sudo lsb_release -a

设置代理：system setting->network->networkproxy->manual->apply system wide

设置apt源：/etc/apt/source.list (ubuntu 14.04版本使用了163镜像)

#install ssh

apt-get  apt-get install openssh-server 

#install java

sudo /opt/java

tar -zxvf ./jdk-7u45-linux-x64.tar.gz

mv jdk1.7.0_45/ jdk1.7/

cp .bashrc .bash_original

vi ~/.bashrc

export JAVA_HOME=/opt/Java/jdk/jdk1.7

export CLASSPATH=${JAVA_HOME}/lib

export PATH=${JAVA_HOME}/bin:$PATH

source ~/.bashrc

#shh 无密码验证登录

ssh-keygen -t rsa -P ""

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

 

#添加hadoop用户

sudo useradd -m hadoop -s -/bin/bash

修改密码:sudo passwd hadoop

增加管理员的权限:sudo adduser hadoop sudo

hadoop用户赋予和root一样的权限：

sudo gedit /etc/sudoers

然后编辑增添如下那句hadoop   ALL(ALL:ALL)  ALL

配置全局java

sudo vi /etc/profile

export JAVA_HOME=/opt/java/jdk1.7

export PATH=$JAVA_HOME/bin:$PATH

export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile

 

 

4. 安装hadoop

4.1. 解压：

sudo tar xzf hadoop-2.7.1.tar.gz（一定不要下载rpm的，这个是redhat等系统的软件安装包，在ubuntu中是tar.gz这种才行。）

4.2. copy到/usr/local/中，并命名为hadoop

sudo mv hadoop-2.7.1 /usr/local/

cd /usr/local

sudo mv hadoop-2.7.1 hadoop

4.3. 修改目录主人

sudo chown hadoop:hadoop -R -f /usr/local/hadoop/

4.4. 配置

打开配置文件:

sudo vim /etc/profile

添加以下命令语句：

export HADOOP_INSTALL=/usr/local/hadoop

export PATH=$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin

export HADOOP_MAPRED_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_HOME=$HADOOP_INSTALL

export HADOOP_HDFS_HOME=$HADOOP_INSTALL

export YARN_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

 

source /etc/profile

4.5.测试

调试

export HADOOP_ROOT_LOGGER=DEBUG,console

 

cp README.txt input

hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.5.1-sources.jar org.apache.hadoop.examples.WordCount input output

 

本地库报错

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

如果你安装的是hadoop 2.4以及之前的版本，它自带的native库是32位的，如果你的操作系统和JVM是64位的，就会报上述错误；

如果你安装的是hadoop 2.5版本，它自带的native库是64位的，如果你的操作系统和JVM是32位的，就会报上述错误。

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib/native"

或者 export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

加载成功：

15/12/25 17:47:50 DEBUG util.NativeCodeLoader: Trying to load the custom-built n

ative-hadoop library...

15/12/25 17:47:50 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library

 

hadoop 编译

安装maven

sudo apt-get install maven

mvn --version

安装依赖库

sudo apt-get install g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev

安装protoc 

sudo apt-get install protobuf-compiler

protoc --version

解压

tar -zxvf hadoop-2.5.1-src.tar.gz

编译

mvn package -Pdist,native -DskipTests -Dtar 

编译报错，不能下载插件，原因mvn不使用系统配置的代理，修改setting.xml

Downloading: http://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/2.4.0/maven-bu