hadoop详细安装和配置

hadoop版本：hadoop-2.2.0-cdh5.0.0-beta-1
jdk版本：jdk-7u40-linux-x64

环境准备：
10.95.3.100 master1
10.95.3.101 master2
10.95.3.103 slave1
10.95.3.104 slave2

说明：本安装没有进行hdfs HA配置，NameNode和SecondaryNameNode都在master1上，其他三个节点作为DN节点。

安装步骤（JDK的安装省略）：

1、添加hadoop用户
   root身份登录master1
   添加用户：useradd –d /home/hadoop -m hadoop
   给hadoop用户设置密码：passwd hadoop

2、配置hosts
   以root用户登录各台机器，vi /etc/hosts文件，修改如下：

#127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.95.3.100              master1
10.95.3.101              master2
10.95.3.103              slave1
10.95.3.104              slave2

3、配置master1到各台机器的免密码登录
   以hadoop用户登录master1，进入/home/hadoop目录下，执行:
   ssh-keygen -t rsa
   注意一定不要输入密码，这样，在/home/hadoop/.ssh目录下将会生成id_rsa，  id_rsa.pub两个文件
   然后再分别执行：
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master1
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master2
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave1
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave2
这样，就把master1的公钥加入到了master的authorized_keys文件中。可以测试下从master1 ssh到其他各台机器是否不需要输入密码。
   注意：文件夹的权限
   sudo chmod 755 .ssh/
   sudo chmod 644 .ssh/authorized_keys

4、解压hadoop-2.2.0-cdh5.0.0-beta-1.tar.gz，设置hadoop的安装目录为/dp/hadoop
   设置环境变量：vi ~/.bashrc

export HADOOP_HOME=/dp/hadoop
export HADOOP_PID_DIR=/dp/hadoop_pid_dir
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=/dp/hadoop
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$ANT_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$FLUME_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:.

5、进入/dp/hadoop/etc/hadoop目录，修改配置文件
core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master1</value>
        </property>

        <property>
                <name>fs.trash.interval</name>
                <value>10080</value>
        </property>
        <property>
                <name>fs.trash.checkpoint.interval</name>
                <value>10080</value>
        </property>
        <property>
                <name>topology.script.file.name</name>
                <value>/dp/hadoop/etc/hadoop/rack.py</value>
                </property>
        <property>
                <name>topology.script.number.args</name>
                <value>6</value>
        </property>
        <property>
               <name>hadoop.security.group.mapping</name>
               <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>
        </property>
<!--
        <property> 
                <name>hadoop.native.lib</name> 
                <value>false</value> 
                <description>Should native hadoop libraries, if present, be used.</description> 
        </property>
-->
<!--
        <property>
                <name>ha.zookeeper.quorum</name>
                <value>master:2181,slave2:2181,slave6:2181</value>
        </property>      
        
        <property>
                <name>hadoop.proxyuser.hadoop.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.hadoop.groups</name>
                <value>*</value>
        </property>
-->
</configuration>

hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
                <name>dfs.blocksize</name>
                <value>16m</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/dp/data/hadoop</value>
        </property>

        <property>
                <name>dfs.namenode.http-address</name>
                <value>master1:50070</value>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>master1:50090</value>
        </property>
        <property>
                <name>dfs.webhdfs.enabled</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.datanode.max.xcievers</name>
                <value>1000000</value>
        </property>
        <property>
                <name>dfs.balance.bandwidthPerSec</name>
                <value>104857600</value>
                <description>
                        Specifies the maximum amount of bandwidth that each datanode
                        can utilize for the balancing purpose in term of
                        the number of bytes per second.
                </description>
        </property>
        <property>
                <name>dfs.hosts.exclude</name>
                <value>/dp/hadoop/etc/hadoop/excludes</value>
                <description>Names a file that contains a list of hosts that are
                        not permitted to connect to the namenode.  The full pathname of the
                        file must be specified.  If the value is empty, no hosts are
                        excluded.</description>
         </property>

上述的配置要手工创建excludes文件。

map-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>master1:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master1:19888</value>
        </property>

<!--        <property>
                <name>mapreduce.history.server.delegationtoken.renewer</name>
                <value>true</value>
        </property>
-->
        <property>
                <name>mapreduce.output.fileoutputformat.compress</name>
                <value>true</value>
        </property>
        <property>
                <name>mapreduce.output.fileoutputformat.compress.type</name>
                <value>BLOCK</value>
        </property>
        <property>
                <name>mapreduce.output.fileoutputformat.compress.codec</name>
                <value>org.apache.hadoop.io.compress.SnappyCodec</value>
        </property>
        <property>
                <name>mapreduce.map.output.compress</name>
                <value>true</value>
        </property>
        <property>
        <property>
                <name>mapreduce.map.output.compress.codec</name>
                <value>org.apache.hadoop.io.compress.SnappyCodec</value>
        </property>

yarn-site.xml

<configuration>
        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>master1:8031</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>master1:8032</value>
        </property>
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>master1:8030</value>
        </property>

        <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>master1:8033</value>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>master1:8088</value>
        </property>
        <property>
                <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
                <value>10000</value>
        </property>
        <property>
                <description>Classpath for typical applications.</description>
                <name>yarn.application.classpath</name>
                <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
                        $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
                        $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
                        $YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,
                        $YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
                <name>yarn.nodemanager.local-dirs</name>
                <value>/dp/data/yarn/local</value>
        </property>
        <property>
                <name>yarn.nodemanager.log-dirs</name>
                <value>/dp/data/yarn/logs</value>
        </property>
        <property>
                <description>Where to aggregate logs</description>
                <name>yarn.nodemanager.remote-app-log-dir</name>
                <value>/dp/data/yarn/logs</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.staging-dir</name>
                <value>/user</value>
        </property>
        <property>
                <description>Amount of physical memory, in MB, that can be allocated
                  for containers.</description>
                <name>yarn.nodemanager.resource.memory-mb</name>
                <value>2048</value>
        </property>

6、配置机架感知,在/dp/hadoop/etc/hadoop下添加rack.data rack.py文件
rack.data

default /rack/default

10.95.3.101 master2 /rack/rack1
10.95.3.103 slave1  /rack/rack1
10.95.3.104 slave2  /rack/rack1

rack.py

#!/bin/env python

import sys,os,time

pwd = os.path.realpath( __file__ )
rack_file = os.path.dirname(pwd) + "/rack.data"

rack_list = [ l.strip().split() for l in open(rack_file).readlines() if len(l.strip().split()) > 1 ]
rack_map = {}
for item in rack_list:
        for host in item[:-1]:
                rack_map[host] = item[-1]
rack_map['default'] = 'default' in rack_map and rack_map['default'] or '/default/rack'
rack_result = [av in rack_map and rack_map[av] or rack_map['default'] for av in sys.argv[1:]]
#print rack_map, rack_result
print ' '.join( rack_result )

f = open('/tmp/rack.log','a+')
f.writelines( "[%s] %sn" % (time.strftime("%F %T"),str(sys.argv)))
f.close()

7、修改slaves文件：

master2
slave1
slave

8、将master1上配置好的hadoop拷贝到其他节点上去

9、执行start-dfs.sh启动hadoop，注意如果防火墙没关闭，有可能造成DN连接不上NN

10、执行hadoop dfsadmin -report命令，查看hadoop是否正常启动