【转载】Hadoop2.6.0 mac 安装

1. 单机模式

(1) ssh设置远程登录：

xiaolan:~ xiaolan$@ssh-keygen -t rsa -P ""

xiaolan:~ xiaolan$cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

登录测试 

xiaolan:~ xiaolan$ssh localhost

xiaolan:~ xiaolan$exit

(2) 安装jdk(忽略)

(3) 配置环境变量

xiaolan:~ xiaolan$vim ~/.bash_profile

输入内容：

export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0

export YARN_HOME=/Users/xiaolan/Applications/hadoop-2.6.0

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$YARN_HOME/etc/hadoop

:wq

让环境变量生效

xiaolan:~ xiaolan$source ~/.bash_profile

(4) 配置hadoop

进入Applications/hadoop-2.6.0/etc/hadoop

xiaolan:~ xiaolan$vim hadoop-env.sh

输入内容：
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home

export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0

export PATH=$PATH:/Users/xiaolan/Applications/hadoop-2.6.0/bin

:wq

让环境变量生效

xiaolan:~ xiaolan$source hadoop-env.sh

单机模式安装成功

(5) 测试

xiaolan:~ xiaolan$mkdir input

xiaolan:~ xiaolan$cp **/** input

注：随便拷贝点文件

xiaolan:~ xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output

xiaolan:~ xiaolan$cat output/*

注：成功看到结果

2. 伪分布模式

(1) 配置core-site.xml

<configuration></configuration>之间增加如下内容：

<property>

        <name>fs.default.name</name>

        <value>hdfs://localhost:9000</value>

</property>

(2) 配置yarn-site.xml

<configuration></configuration>之间增加如下内容： 

<property> 

<name>yarn.noCHdemanager.aux-services</name> 

<value>mapreduce_shuffle</value> 

</property> 

 <property> 

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value> 

</property>

(3) 创建和配置mapred-site.xml

默认/etc/hadoop文件夹下有mapred.xml.template文件，复制更改用户名mapred.xml

<configuration></configuration>之间增加如下内容： 

<property> 

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

(4) 配置hdfs-site.xml

指定主机上作为namenode和datanode的目录：

/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name

/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data

<configuration></configuration>之间增加如下内容：

<property>

<name>dfs.replication</name>

        <value>1</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data</value>

</property>

(5) 格式化hdfs

xiaolan:~ xiaolan$hdfs namenode -format

(6) 启动hadoop

xiaolan:~ xiaolan$cd Applications/hadoop-2.6.0/sbin

xiaolan:sbin xiaolan$./start-dfs.sh

xiaolan:sbin xiaolan$./start-yarn.sh

浏览器打开 http://localhost:50070/，会看到hdfs管理页面

浏览器打开 http://localhost:8088/，会看到hadoop进程管理页面

(7) WordCount验证

创建input目录

xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -mkdir input

fs:hadoop下的文件命令，和普通命令基本一致

拷贝文件

xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -copyFromLocal README.txt input

xiaolan:hadoop-2.6.0 xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output

xiaolan:hadoop-2.6.0 xiaolan$cat output/*

注：成功看到结果

 

 

linux：图文教程，版本较老：

http://www.linuxidc.com/Linux/2013-01/77681p4.htm

http://www.cnblogs.com/kinglau/p/3794433.html

官网安装教程：

http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html

 

Eclipse配置：

http://andy-ghg.iteye.com/blog/1165453

 

注：

据说可以忽略：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable：

http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos