hadoop安装步骤

1.安装jdk（自带了jre），单纯的jre是不够的，MapReduce的编写和Hadoop的编译都依赖于jdk。(注意jdk必须1.6以上，本教程使用的是jdk1.6.0_24)

2、安装cygwin，下载地址 http://www.cygwin.com,本教程采用的是1.7.9

     必须选择的安装包： Net Category----openssh和openssl

      可选包：Editors Category---vim，方便修改配置文件

                      Devel  Category----subversion

3. 配置cygwin的环境变量

点击桌面的cygwin图标，启动cygwin，执行如下命令
    $vim /etc/profile

   在最后添加如下行：

export JAVA_HOME=/cygdrive/d/Java/jdk1.6.0_24
export HADOOP_HOME=/cygdrive/d/hadoop-0.21.0
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

4.在cygwin中安装sshd

点击桌面的cygwin图标，启动cygwin，执行如下命令

$ssh-host-config,当要求输入yes/no时，选择no。当看到“Have fun”，一般表示sshd服务安装成功。

5.启动sshd服务

在桌面上的“我的电脑”图标上单击右键，点击“管理”，在打开的窗口左侧菜单中选择“服务和应用程序”，在右侧的列表中，在sshd行上点击右键，选择“启动”

6.配置ssh登录

点击桌面的cygwin图标，启动cygwin，执行如下命令

#生成密钥文件

$ssh-keygen   一路回车即可

#加为信任证书

$cd ~/.ssh

$cp id_rsa.pub authorized_keys

#exit

重新点击桌面的cygwin图标，启动cygwin

$ssh localhost
如果不再提示输入密码，则成功

7.下载hadoop-0.21.0的安装包，hadoop-0.21.0.tar.gz

8.安装hadoop

将hadoop-0.21.0.tar.gz 解压，如解压到d:\hadoop-0.21.0

配置conf下的hadoop-env.sh ,zhi
只需要将JAVA_HOME修改成jdk的安装路径即可（注意此处的路径不是windows风格的目录d:\java\dk1.6.0_24,而是linux风格/cygwin/d/java/jdk1.6.0_24)

配置conf下的mapred.site ，增加如下代码

<configuration>
    <property>
            <name>fs.default.name</name>
            <value>hdfs://localhost:9000</value>
    </property>   
    <property>
            <name>mapred.job.tracker</name>
            <value>hdfs://localhost:9001</value>
    </property>
  <property>
            <name>dfs.replication</name>
            <value>1</value>
    </property>
</configuration>

9. format a new distributed-files system

$bin/hadoop namecode   -format

10.start the hadoop daemons

$bin/start-all.sh

11.browser the web interface for NameCode and JobTracker,by default they are available at
NameCode- http://localhost:50070

JobTracker-http://localhost:50030

12.common Exception

12.1解决Window环境下启动Hadoop时出现的 java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName 异常在Window下启动Hadoop-0.21.0版本时，会出现下面这样的错误提示：

1 java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
2 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
3
4         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
5         at java.security.AccessController.doPrivileged(Native Method)
6         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
7         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
8         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
9         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
10 Could not find the main class: org.apache.hadoop.util.PlatformName.  Program wil
11 l exit.

经过不断的查找原因和尝试，终于有了解决这个错误的办法，只需要将${HADOOP_HOME}/bin/hadoop-config.sh文件中的第190行的一下的内容

JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
12.2hadoop无法正常启动(1)
执行  $ bin/hadoop start-all.sh之后,无法启动.

异常一

Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.

localhost:      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:214)

localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:135)

localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:119)

localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:481)


解决方法：
此时是没有配置conf/mapred-site.xml的缘故.  在0.21.0版本上是配置mapred-site.xml,在之前的版本是配置core-site.xml,0.20.2版本中配置mapred-site.xml无效,只能配置core-site.xml文件
<configuration>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://localhost:9000</value>

  </property>

  <property>

    <name>mapred.job.tracker</name>

    <value>hdfs://localhost:9001</value>

  </property>

  <property>

    <name>dfs.replication</name>

    <value>1</value>

  </property>

</configuration>



hadoop无法正常启动(2)

异常二、

starting namenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-namenode-aist.out

localhost: starting datanode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-datanode-aist.out

localhost: starting secondarynamenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-secondarynamenode-aist.out

localhost: Exception in thread "main" java.lang.NullPointerException

localhost:      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)

localhost:      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)

localhost:      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)

localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131)

localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:115)

localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469)

starting jobtracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-jobtracker-aist.out

localhost: starting tasktracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-tasktracker-aist.out



解决方法：
此时是没有配置conf/mapred-site.xml的缘故.  在0.21.0版本上是配置mapred-site.xml,在之前的版本是配置core-site.xml   , 0.20.2版本中配置mapred-site.xml无效,只能配置core-site.xml文件
<configuration>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://localhost:9000</value>

  </property>

  <property>

    <name>mapred.job.tracker</name>

    <value>hdfs://localhost:9001</value>

  </property>

  <property>

    <name>dfs.replication</name>

    <value>1</value>

  </property>

</configuration>



hadoop无法正常启动(3)

异常三、

starting namenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-namenode-aist.out

localhost: starting datanode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-datanode-aist.out

localhost: Error: JAVA_HOME is not set.

localhost: starting secondarynamenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-secondarynamenode-aist.out

localhost: Error: JAVA_HOME is not set.

starting jobtracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-jobtracker-aist.out

localhost: starting tasktracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-tasktracker-aist.out

localhost: Error: JAVA_HOME is not set.



解决方法:

请在$hadoop/conf/hadoop-env.sh文件中配置JDK的环境变量

JAVA_HOME=/home/xixitie/jdk

CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME CLASSPATH




hadoop无法正常启动(4)
异常四:mapred-site.xml配置中使用hdfs://localhost:9001，而不使用localhost:9001的配置

异常信息如下：

11/04/20 23:33:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.sec                                                                             urity.ShellBasedUnixGroupsMapping; cacheTimeout=300000

11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste                                                                             m name. Use "hdfs://localhost:9000/" instead.

11/04/20 23:33:25 WARN conf.Configuration: mapred.task.id is deprecated. Instead                                                                             , use mapreduce.task.attempt.id

11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste                                                                             m name. Use "hdfs://localhost:9000/" instead.

11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste                                                                             m name. Use "hdfs://localhost:9000/" instead.



解决方法：

mapred-site.xml配置中使用hdfs://localhost:9000，而不使用localhost:9000的配置

  <property>

    <name>fs.default.name</name>

    <value>hdfs://localhost:9000</value>

  </property>

  <property>

    <name>mapred.job.tracker</name>

    <value>hdfs://localhost:9001</value>

  </property>



hadoop无法正常启动(5)

异常五、no namenode to stop 问题的解决:

异常信息如下：
11/04/20 21:48:50 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 0 time(s).

11/04/20 21:48:51 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 1 time(s).

11/04/20 21:48:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 2 time(s).

11/04/20 21:48:53 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 3 time(s).

11/04/20 21:48:54 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 4 time(s).

11/04/20 21:48:55 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 5 time(s).

11/04/20 21:48:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 6 time(s).

11/04/20 21:48:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 7 time(s).

11/04/20 21:48:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 8 time(s).



解决方法:

这个问题是由namenode没有启动起来引起的,为什么no namenode to stop,可能之前的一些数据对namenode有影响,

你需要执行:

$ bin/hadoop namenode -format

然后

$bin/hadoop start-all.sh



hadoop无法正常启动(6)

异常五、no datanode to stop 问题的解决:

有时数据结构出现问题会产生无法启动datanode的问题。

然后用 hadoop namenode -format  重新格式化后仍然无效，/tmp中的文件并没有清楚。

其实还需要清除/tmp/hadoop*里的文件。

执行步骤：

     一、先删除hadoop:///tmp

       hadoop  fs -rmr /tmp

    二、停止 hadoop  

       stop-all.sh

    三、删除/tmp/hadoop*

       rm -rf /tmp/hadoop*

    四、格式化hadoop

       hadoop namenode -format

    五、启动hadoop

        start-all.sh

之后即可解决这个datanode没法启动的问题


http://blog.csdn.net/dream8062/article/details/7281744