hadoop0.20.2完全分布式安装和配置

                                  苦一时，甜一辈子啊亲。。。刚巴德

                                              完全分布式安装和配置
      

 硬件环境：

       虚拟机1号：master   hostname: h1    IP:192.168.1.106    ==》用户名：hadoop

       虚拟机2号：slave1    hostname:  h2   IP:192.168.1.107    ==》用户名：hadoop

       虚拟机3号    slave2    hostname:  h3   IP:192.168.1.108    ==》用户名：hadoop

 

1.配置hosts(三台机器都需要操作)

     [root@hadoop /]#  vi /etc/hosts

     192.168.1.106  h1

     192.168.1.107  h2

     192.168.1.108  h3

2.建立统一的hadoop用户

     *******

3.设置ssh免登陆密码（生成秘钥对）

    键入：cd ~/.ssh

    键入：ssh-keygen -t  rsa

   生成两个文件：id_rsa  id_rsa.pub

   然后将id_rsa.pub复制一份至authorized_keys中：

   键入：cat id_rsa.pub >> authorized_keys

   同时在上个虚拟机上做同样的事情，让后将所有的虚拟机上的authorized_keys内容合并，将合并后的

   内容在分发到每个虚拟机上

4.检验一下是否可以免密码登录到其他虚拟机上(免登陆第一次需要写密码)

   键入：ssh h1  然后输入密码若能登录说明配置成功了

5.下载hadoop-0.20.2.tar.gz解压

   我是解压在/home/soft/hadoop下面在

6.修改相关配置文件

hadoop-env.sh文件修改内容

# Set Hadoop-specific environment variables here.

 

# The only required environment variable is JAVA_HOME.  All others are

# optional.  When running a distributed configuration it is best to

# set JAVA_HOME in this file, so that it is correctly defined on

# remote nodes.

 

# The java implementation to use.  Required.

 export JAVA_HOME=/usr/java/jdk1.8.0_45

 

# Extra Java CLASSPATH elements.  Optional.

# export HADOOP_CLASSPATH=

 

# The maximum amount of heap to use, in MB. Default is 1000.

# export HADOOP_HEAPSIZE=2000

 

# Extra Java runtime options.  Empty by default.

# export HADOOP_OPTS=-server

 

# Command specific options appended to HADOOP_OPTS when specified

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"

export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"

export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"

# export HADOOP_TASKTRACKER_OPTS=

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)

# export HADOOP_CLIENT_OPTS

 

# Extra ssh options.  Empty by default.

# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"

 

# Where log files are stored.  $HADOOP_HOME/logs by default.

# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

 

# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.

# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

 

# host:path where hadoop code should be rsync'd from.  Unset by default.

# export HADOOP_MASTER=master:/home/$USER/src/hadoop

 

# Seconds to sleep between slave commands.  Unset by default.  This

# can be useful in large clusters, where, e.g., slave rsyncs can

# otherwise arrive faster than the master can service them.

# export HADOOP_SLAVE_SLEEP=0.1

 

# The directory where pid files are stored. /tmp by default.

# export HADOOP_PID_DIR=/var/hadoop/pids

 

# A string representing this instance of hadoop. $USER by default.

# export HADOOP_IDENT_STRING=$USER

 

# The scheduling priority for daemon processes.  See 'man nice'.

# export HADOOP_NICENESS=10

 

 

 

core-site.xml文件修改内容

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

 

 

 

hdfs-site.xml文件修改内容

    <?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

 

<property>

<name>dfs.data.dir</name>

<value>/home/soft/hadoop/hadoop-0.20.2/data</value><!-- 数据节点路径 -->

</property>

 

<property>

<name>dfs.replication</name>

<value>1</value> <!-- 数据块复制多少份 -->

</property>

 

</configuration>

 

 

 mapred-site.xml文件修改内容
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

7.格式化分布式文件系统

[root@hadoop hadoop-0.20.2]# bin/hadoop namenode -format
15/08/03 23:57:29 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop/192.168.1.103
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
15/08/03 23:57:30 INFO namenode.FSNamesystem: fsOwner=root,root
15/08/03 23:57:30 INFO namenode.FSNamesystem: supergroup=supergroup
15/08/03 23:57:30 INFO namenode.FSNamesystem: isPermissionEnabled=true
15/08/03 23:57:30 INFO common.Storage: Image file of size 94 saved in 0 seconds.
15/08/03 23:57:30 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
15/08/03 23:57:30 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop/192.168.1.103
************************************************************/

8.启动守护进程

[root@hadoop hadoop-0.20.2]# bin/start-all.sh

starting namenode, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-hadoop.out

 

The authenticity of host 'localhost (::1)' can't be established.

 

RSA key fingerprint is 9d:53:31:22:f4:06:51:7c:9c:95:bc:91:0b:1b:8f:82.

 

Are you sure you want to continue connecting (yes/no)? yes

 

localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

 

root@localhost's password: 

 

localhost: starting datanode, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-datanode-hadoop.out

 

root@localhost's password: 

 

localhost: starting secondarynamenode, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-secondarynamenode-hadoop.out

 

starting jobtracker, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-hadoop.out

 

root@localhost's password: 

 

localhost: starting tasktracker, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-tasktracker-hadoop.out

9.观看与java有关的进程

[root@hadoop hadoop-0.20.2]# ls /usr/java/jdk1.8.0_45/bin/jps

 

/usr/java/jdk1.8.0_45/bin/jps

 

[root@hadoop hadoop-0.20.2]# /usr/java/jdk1.8.0_45/bin/jps

 

14112 NameNode

 

14384 JobTracker

 

14209 DataNode

 

14483 TaskTracker

 

14329 SecondaryNameNode

 

14638 Jps

 

 

最后要干的事情就是将刚刚解压后的hadoop-0.20.2文件夹整个拷贝到另外两个slave机器下面。

然后执行bin/stop-all.sh关闭所有的进程，再启动bin/start-all.sh