hadoop 集群配置

1.下载hadoop version 0.20.205.0
2.下载jdk1.6 version 1.6.0_29
3.准备4台机器

name	ip	type
oplog.master	10.125.14.205	master
oplog.slave1	10.125.14.195	slave
oplog.slave2	10.125.14.19	slave
oplog.slave3	10.125.14.176	slave

4.修改/etc/profile,增加如下配置(四台机器均加入)

..
export JAVA_HOME=/home/jincheng.sunjc/softs/jdk1.6.0_29/
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:
..

[jincheng.sunjc@v014205-bja hadoop20205]$ source /etc/profile

5.修改/etc/hosts,增加如下配置(四台机器均加入)

10.125.14.205 oplog.master
10.125.14.195 oplog.slave1
10.125.14.19 oplog.slave2
10.125.14.176 oplog.slave3

[jincheng.sunjc@v014205-bja hadoop20205]$ source /etc/profile

6.修改../hadoop20205/conf/hadoop-env.sh

export JAVA_HOME=/home/jincheng.sunjc/softs/jdk1.6.0_29/
expor tHADOOP_HOME_WARN_SUPPRESS=TRUE

7.修改../hadoop20205/conf/core-site.xml

<configuration>
<property>
<name></name>
<value>/home/jincheng.sunjc/softs/hadoop/data</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://10.125.14.205:9000</value>
</property>
</configuration>

8.修改hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>

9.修改mapred-site.xml

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://10.125.14.205:9001</value>
</property>
</configuration>

10.配置修改../hadoop20205/conf/masters

oplog.master

11.配置修改../hadoop20205/conf/slaves

oplog.slave1
oplog.slave2
oplog.slave3

12.拷贝master机器的目录到slave1 slave2 slave3

scp -r softs/ slave1:/home/jincheng.sunjc/.
scp -r softs/ slave2:/home/jincheng.sunjc/.
scp -r softs/ slave3:/home/jincheng.sunjc/.

13.配置master本地免登

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

14.配置master和slave1 slave2 slave3 免登

master：scp ~/.ssh/id_dsa.pub slave1:/home/jincheng.sunjc/.
slave1：cat /home/jincheng.sunjc/id_dsa.pub >> ~/.ssh/authorized_keys
slave2和slave3同样操作

15.启动集群

[jincheng.sunjc@v014205-bja bin]$ ./hadoop namenode -format
13/04/22 16:50:39 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = v014205-bja/10.125.14.205
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.205.0
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205 -r 1179940; compiled by 'hortonfo' on Fri Oct  7 06:20:32 UTC 2011
************************************************************/
Re-format filesystem in /tmp/hadoop-jincheng.sunjc/dfs/name ? (Y or N) Y
13/04/22 16:50:41 INFO util.GSet: VM type       = 64-bit
13/04/22 16:50:41 INFO util.GSet: 2% max memory = 19.33375 MB
13/04/22 16:50:41 INFO util.GSet: capacity      = 2^21 = 2097152 entries
13/04/22 16:50:41 INFO util.GSet: recommended=2097152, actual=2097152
13/04/22 16:50:41 INFO namenode.FSNamesystem: fsOwner=jincheng.sunjc
13/04/22 16:50:41 INFO namenode.FSNamesystem: supergroup=supergroup
13/04/22 16:50:41 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/04/22 16:50:41 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/04/22 16:50:41 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/04/22 16:50:41 INFO namenode.NameNode: Caching file names occuring more than 10 times 
13/04/22 16:50:42 INFO common.Storage: Image file of size 120 saved in 0 seconds.
13/04/22 16:50:42 INFO common.Storage: Storage directory /tmp/hadoop-jincheng.sunjc/dfs/name has been successfully formatted.
13/04/22 16:50:42 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at v014205-bja/10.125.14.205
************************************************************/

[jincheng.sunjc@v014205-bja bin]$ ./start-all.sh 
starting namenode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-namenode-v014205-bja.out
oplog.slave3: starting datanode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-datanode-v014176-bja.out
oplog.slave1: starting datanode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-datanode-v014195-bja.out
oplog.slave2: starting datanode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-datanode-v014019-bja.out
oplog.master: starting secondarynamenode, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-secondarynamenode-v014205-bja.out
starting jobtracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-jobtracker-v014205-bja.out
oplog.slave1: starting tasktracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-tasktracker-v014195-bja.out
oplog.slave2: starting tasktracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-tasktracker-v014019-bja.out
oplog.slave3: starting tasktracker, logging to /home/jincheng.sunjc/softs/hadoop/hadoop20205/libexec/../logs/hadoop-jincheng.sunjc-tasktracker-v014176-bja.out
[jincheng.sunjc@v014205-bja bin]$ 

16.验证集群启动情况
1）验证master,有namenode jobtracker和secondaryNameNode进程

[jincheng.sunjc@v014205-bja bin]$ jps
4559 Jps
4345 SecondaryNameNode
4443 JobTracker
4170 NameNode
[jincheng.sunjc@v014205-bja bin]$ 

2）验证slave,有dataNode 和 tasktracker进程

[jincheng.sunjc@v014195-bja hadoop]$ jps
32618 DataNode
32718 TaskTracker
330 Jps
[jincheng.sunjc@v014195-bja hadoop]$

3）命令校验