2-集群安装

 

1）准备环境

 

hostname	ip	作用
station1	192.168.80.51	Namenaode jobtracher datanode tasktracher
station2	192.168.80.52	Datanode  jobtracher

用户为：cloud

 

 

2）分别安装HADOOP

分别建立文件

[root@station1 ~]# mkdir /cloud

[root@station1 ~]# chmod 777 /cloud

 

[cloud@station1 cloud]$ pwd

/cloud

[cloud@station1 cloud]$ ll

total 4

drwxr-xr-x 13 cloud cloud 4096 May  9 04:35 hadoop-1.0.3

 

 

修改配制文件 namenode

 

[cloud@station1 conf]$ pwd

/cloud/hadoop-1.0.3/conf

[cloud@station1 conf]$ vi core-site.xml

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

        <property>

                <name>fs.default.name</name>

                <value>hdfs://station1:9000</value>

        </property>

 

</configuration>

 

 

[cloud@station1 conf]$ vi hdfs-site.xml

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

 <property>

         <name>dfs.replication</name>

         <value>1</value>

</property>

 

 

</configuration>

 

 

 

 

[cloud@station1 conf]$ vi /hadoop/hadoop-1.0.3/conf/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_09

 

 

另一个节点也一样配的。

 

 

 

 

下面还要这样配

[cloud@station1 conf]$ more masters

station1

[cloud@station1 conf]$ more slaves

station1

station2

 

 

 

 

 

 

 

3)修改/etc/hosts 文件

~

[root@station1 ~]# vi /etc/hosts

 

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1               station1 localhost.localdomain localhost

192.168.80.51            station1

192.168.80.52            station2

::1             localhost6.localdomain6 localhost6

 

 

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1               station1 localhost.localdomain localhost

192.168.80.51            station1

192.168.80.52            station2

::1             localhost6.localdomain6 localhost6

 

 

 

 

 

4）增加 SSH信任

1）分别修改hadoop 配制文件masters   和  slaves

[root@station2 ~]# useradd cloud

[root@station2 ~]# passwd cloud

Changing password for user cloud.

New UNIX password:

BAD PASSWORD: it is too simplistic/systematic

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

 

cloud@station2 ~]$  ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

Generating public/private dsa key pair.

/home/cloud/.ssh/id_dsa already exists.

Overwrite (y/n)? y

Your identification has been saved in /home/cloud/.ssh/id_dsa.

Your public key has been saved in /home/cloud/.ssh/id_dsa.pub.

The key fingerprint is:

4c:89:95:c5:89:cc:d8:a9:de:ce:e8:8e:7f:13:75:0f cloud@station2

[cloud@station2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[cloud@station2 ~]$ chmod 644 ~/.ssh/authorized_keys

[cloud@station2 ~]$  scp ~/.ssh/authorized_keys station1:/home/cloud/.ssh/

cloud@station1's password:

authorized_keys                                                                                                                                100% 1812     1.8KB/s   00:00   

[cloud@station2 ~]$ ssh station1

Last login: Tue Oct 30 13:44:56 2012 from station2

[cloud@station1 ~]$ hostname

station1

 

 

 

5）启动

在主节点

 

[cloud@station1 conf]$ hadoop namenode -format

Warning: $HADOOP_HOME is deprecated.

 

12/10/30 14:59:03 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = station1/192.168.80.51

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.0.3

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May  8 20:31:25 UTC 2012

************************************************************/

12/10/30 14:59:03 INFO util.GSet: VM type       = 32-bit

12/10/30 14:59:03 INFO util.GSet: 2% max memory = 19.33375 MB

12/10/30 14:59:03 INFO util.GSet: capacity      = 2^22 = 4194304 entries

12/10/30 14:59:03 INFO util.GSet: recommended=4194304, actual=4194304

12/10/30 14:59:04 INFO namenode.FSNamesystem: fsOwner=cloud

12/10/30 14:59:04 INFO namenode.FSNamesystem: supergroup=supergroup

12/10/30 14:59:04 INFO namenode.FSNamesystem: isPermissionEnabled=true

12/10/30 14:59:04 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

12/10/30 14:59:04 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

12/10/30 14:59:04 INFO namenode.NameNode: Caching file names occuring more than 10 times

12/10/30 14:59:04 INFO common.Storage: Image file of size 111 saved in 0 seconds.

12/10/30 14:59:04 INFO common.Storage: Storage directory /tmp/hadoop-cloud/dfs/name has been successfully formatted.

12/10/30 14:59:04 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at station1/192.168.80.51

************************************************************/

 

[cloud@station1 hadoop-1.0.3]$ ./bin/start-all.sh

Warning: $HADOOP_HOME is deprecated.

 

starting namenode, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-namenode-station1.out

station2: starting datanode, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-datanode-station2.out

station1: starting datanode, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-datanode-station1.out

station1: Error: JAVA_HOME is not set.

station2: Error: JAVA_HOME is not set.

station1: starting secondarynamenode, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-secondarynamenode-station1.out

station1: Error: JAVA_HOME is not set.

starting jobtracker, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-jobtracker-station1.out

station2: starting tasktracker, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-tasktracker-station2.out

station1: starting tasktracker, logging to /cloud/hadoop-1.0.3/libexec/../logs/hadoop-cloud-tasktracker-station1.out

station2: Error: JAVA_HOME is not set.

station1: Error: JAVA_HOME is not set.

[cloud@station1 hadoop-1.0.3]$ ./bin/stop-all.sh 


 
 

 

安装完成

 

 

 

 

 

 

下面进行测试

 

测试文件

[cloud@station1 cloud]$ hadoop fs -put eclipse-java-helios-SR2-linux-gtk.tar.gz  /

 上传文件没有问题

 

        集群的安装请参考：http://pftzzg.iteye.com/blog/1910171

   hadoop-0.20 请参考：http://pftzzg.iteye.com/admin/blogs/1911023

               1.03  请参考：http://pftzzg.iteye.com/admin/blogs/1910153