hadoop 安装和配置

一、安装 JDK

$chmod +x jdk-6u38-linux-i586.bin

$./jdk-6u38-linux-i586.bin

安装完后设置java 环境变量如下

命令 ：/home路径下

$vi /etc/profile

然后添加如下语句

export JAVA_HOME=/home/java/jdk1.6.0_38

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

然后source /etc/profile 使其生效

 

二、ssh 的安装和配置

$实现ssh无密码登陆

$sudo apt-get install ssh

$ssh-keygen 

直接回车，完成后会在~/.ssh/生成两个文件：id_dsa 和id_dsa.pub。这两个是成对

出现，类似钥匙和锁。再把id_dsa.pub 追加到授权key 里面(当前并没有authorized_keys 

$cat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys完成后可以实现无密码登录本机

 

linux-6fp1:/useradd hadoop

linux-6fp1:/home # mkdir hadoop

hadoop@linux-6fp1:~> mkdir .ssh

hadoop@linux-6fp1:~> ls

hadoop@linux-6fp1:~> cd .ssh

hadoop@linux-6fp1:~/.ssh> ssh-keygen -t dsa -P '' -f id_dsa

Generating public/private dsa key pair.

Your identification has been saved in id_dsa.

Your public key has been saved in id_dsa.pub.

The key fingerprint is:

32:32:2a:39:a0:8f:c5:6e:f7:59:9b:b9:1a:ef:71:a6 hadoop@linux-6fp1

The key's randomart image is:

+--[ DSA 1024]----+

|                 |

|                 |

|                 |

|                 |

|.   o o S        |

|oo . o o         |

|= +   . o o      |

| B. .  = O       |

|..o. .+oE.       |

+-----------------+

hadoop@linux-6fp1:~/.ssh> ls

id_dsa  id_dsa.pub

hadoop@linux-6fp1:~/.ssh> cat id_dsa.pub >>authorized_keys

hadoop@linux-6fp1:~/.ssh> ll

total 12

-rw-r--r-- 1 hadoop users 607 2012-12-25 14:39 authorized_keys

-rw------- 1 hadoop users 668 2012-12-25 14:38 id_dsa

-rw-r--r-- 1 hadoop users 607 2012-12-25 14:38 id_dsa.pub

hadoop@linux-6fp1:~/.ssh> ssh -version

OpenSSH_5.1p1, OpenSSL 0.9.8h 28 May 2008

Bad escape character 'rsion'.

hadoop@linux-6fp1:~/.ssh> ssh localhost

The authenticity of host 'localhost (::1)' can't be established.

RSA key fingerprint is a4:93:5e:af:9f:52:41:ce:c8:44:8f:46:60:5f:68:95.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

Have a lot of fun...

 

 

三、Hadoop 安装与配置

1.下载Hadoop

   http://labs.mop.com/apache-mirror/hadoop/common/hadoop-1.0.0/

2.解压Hadoop

         tar xzvf hadoop-0.21.0.tar.gz

3. 添加Hadoop Bin到环境变量中

 

修改 hadoop-env.sh

配置JDK即可

export JAVA_HOME=/home/java/jdk1.6.0_38 

 

vi conf/core-site.xml

<configuration>       

    <property>    

          <name>fs.default.name</name>    

          <value>hdfs://master:9000</value>    

    </property>    

    <property>  

           <name>hadoop.tmp.dir</name>  

           <value>/home/hadoop/tmp</value>  

    </property>  

</configuration>    

 

vi conf/hdfs-site.xml

<configuration>       

   <property>    

         <name>dfs.replication</name>    

         <value>1</value>    

    </property>    

</configuration>    

 

vi conf/mapred-site.xml

<configuration>       

     <property>    

         <name>mapred.job.tracker</name>    

         <value>localhost:9001</value>    

    </property>    

</configuration>    

 

vi conf/master

localhost 改为ip 192.168.1.102

 

vi conf/slaves 

localhost 改为  192.168.1.102

 

启动 Hadoop

 1.格式化文件系统

      hadoop namenode –format

 2. 启动hadoop

     启动关闭所有服务 start-all.sh/stop-all.sh

     启动关闭HDFS: start-dfs.sh/stop-dfs.sh

     启动关闭MapReduce:  start-mapred.sh/stop-mapred.sh

 3. 用jps命令查看进程,确保有   namenode,dataNode,JobTracker,TaskTracker

hadoop@linux-w99d:~/hadoop-0.20.2/bin> ./hadoop namenode -format  

12/12/30 14:26:41 INFO namenode.NameNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = master/192.168.1.102

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 0.20.2

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010

************************************************************/

12/12/30 14:26:41 INFO namenode.FSNamesystem: fsOwner=hadoop,users,dialout,video

12/12/30 14:26:41 INFO namenode.FSNamesystem: supergroup=supergroup

12/12/30 14:26:41 INFO namenode.FSNamesystem: isPermissionEnabled=true

12/12/30 14:26:41 INFO common.Storage: Image file of size 96 saved in 0 seconds.

12/12/30 14:26:42 INFO common.Storage: Storage directory /home/hadoop/tmp/dfs/name has been successfully formatted.

12/12/30 14:26:42 INFO namenode.NameNode: SHUTDOWN_MSG: 

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.102

************************************************************/

hadoop@linux-w99d:~/hadoop-0.20.2/bin> ./start-all.sh

starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-linux-w99d.out

192.168.1.102: datanode running as process 5354. Stop it first.

192.168.1.102: secondarynamenode running as process 5463. Stop it first.

jobtracker running as process 5530. Stop it first.

192.168.1.102: tasktracker running as process 5644. Stop it first.

hadoop@linux-w99d:~/hadoop-0.20.2/bin> jps

6229 NameNode

5644 TaskTracker

6536 Jps

5463 SecondaryNameNode

5530 JobTracker

5354 DataNode

hadoop@linux-w99d:~/hadoop-0.20.2/bin> ./hadoop dfsadmin -report

Configured Capacity: 138087571456 (128.6 GB)

Present Capacity: 130494750735 (121.53 GB)

DFS Remaining: 130494726144 (121.53 GB)

DFS Used: 24591 (24.01 KB)

DFS Used%: 0%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

 

-------------------------------------------------

Datanodes available: 1 (1 total, 0 dead)

 

Name: 192.168.1.102:50010

Decommission Status : Normal

Configured Capacity: 138087571456 (128.6 GB)

DFS Used: 24591 (24.01 KB)

Non DFS Used: 7592820721 (7.07 GB)

DFS Remaining: 130494726144(121.53 GB)

DFS Used%: 0%

DFS Remaining%: 94.5%

Last contact: Sun Dec 30 14:28:08 CST 2012