hadoop集群配置

hadoop部署：

登录用户：suse
HOSTNAME:server0

4台机器：
192.168.2.10  server0(namenode)
192.168.2.11  server1(datanode)
192.168.2.12  server2(datanode)
192.168.2.13  server3(datanode)

1.首先强调的是 确保各个机器节点之间(任意两台机器之间可以ping IP/主机名 可以ping通)
  ping  IP
  ping  主机名
 
2.修改server0的 /etc/hosts 配置文件
  vi /etc/hosts  在 localhost的下面添加(位置任意)
 
  127.0.0.1       localhost
  #hadoop master
  192.168.2.10    server0
  192.168.2.11    server1
  192.168.2.12    server2
  192.168.2.13    server3
 
  **修改 server0  (默认可能为localhost)
 
3.修改server1,server2,server3的 /etc/hosts 配置文件
  (server1,server2,server3)均如下所示

  127.0.0.1       localhost
  192.168.2.11    server1
  192.168.2.12    server2
  192.168.2.13    server3

  **   **修改 server1/server2/server3  (默认可能为localhost)

4.SSH设置
  1>所有节点生成RSA密钥对
    ssh-keygen -t rsa(拷贝执行)
 
  2>一直回车 默认保存路径为/home/suse/.ssh/下
  3>将master的生成的id_rsa,id_rsa.pub 写入authorized_keys
   
    cat id_rsa.pub >> authorized_keys
    cat id_rsa >> authorized_keys
   
  4>将所有slave的 id_rsa.pub 都写入 master的 authorized_keys,最后将master下的 authorized_keys 分发到所有slave
 
   scp /home/suse/.ssh/authorized_keys  server1:/home/suse/.ssh/
   scp /home/suse/.ssh/authorized_keys  server2:/home/suse/.ssh/
   scp /home/suse/.ssh/authorized_keys  server3:/home/suse/.ssh/
  
   之后通过ssh 各主机名称 验证是否通过
  
   最后，在所有机器执行用户权限命令chmod，命令如下
   chmod 644 authorized_keys
  第一次连接需要密码，输入yes和机器密码就可以。以后即不用再输入
 
5.配置文件

   core-site.xml
    <property>
          <name>fs.default.name</name>
          <value>hdfs://server0:9000</value>
          <description>The name of the default file system. A URI whose
                  scheme and authority determine the FileSystem implementation. The
                  uri's scheme determines the config property (fs.SCHEME.impl) naming
                  the FileSystem implementation class. The uri's authority is used to
                  determine the host, port, etc. for a filesystem.</description>
  </property>

    <property>
          <name>dfs.datanode.socket.write.timeout</name>
          <value>0</value>
          <description>A base for other temporary directories.</description>
   </property>

 
  fs.default.name:hadoop 文件系统路径 (配置文件中一定要用主机名。如果用ip的话 以后各节点通信会不能正确解析)
  dfs.datanode.socket.write.timeout  防止socket 异常
  ----------------------------------------------------------------------------------------------------
  hdfs-site.xml
 
    <property>
        <name>dfs.name.dir</name>
        <value>/server/bin/hadoop/name</value>
        <description>
                hadoop文件元数据空间
        </description>
    </property>

  <property>
          <name>dfs.http.address</name>
          <value>192.168.2.10:50070</value>
          <description>
              NameNode HTTP状态监视地址
          </description>
  </property>


  <property>
          <name>hadoop.tmp.dir</name>
          <value>/server/bin/hadoop/temp</value>
          <description>
              hadoop临时目录
          </description>
  </property>

  <property>
          <name>dfs.data.dir</name>
          <value>/server/bin/hadoop/data</value>
          <description>
              N数据本地目录
          </description>
  </property>


  <property>
          <name>dfs.replication</name>
          <value>2</value>
          <description>
              复本数
          </description>

  </property>

-----------------------------------------------------------------------------------------
    mapred-site.xml
    <property>
              <name>mapred.job.tracker</name>
              <value>server0:9001</value>
              <description>The host and port that the MapReduce job tracker runs
                      at. If "local", then jobs are run in-process as a single map
                      and reduce task.
              </description>
      </property>

      <property>
              <name>mapred.map.tasks</name>
              <value>8</value>
              <description>The default number of map tasks per job.
                      Ignored when mapred.job.tracker is "local".
              </description>
      </property>

      <property>
              <name>mapred.reduce.tasks</name>
              <value>8</value>
              <description>The default number of map tasks per job
                     Ignored when mapred.job.tracker is "local".
              </description>
      </property>

      <property>

              <name>mapred.local.dir</name>

              <value>/server/bin/hadoop/mapred/local</value>

              <description>tasktracker上执行mapreduce程序时的本地目录</description>

      </property>

      <property>
              <name>mapred.system.dir</name>
              <value>/tmp/hadoop/mapred/system</value>

      </property>
     
  -----------------------------------------------------------------------------------------------------------------
  master配置
 
   localhost
   server0
  
  ------------------------------------------------------------------------------------------------------------------
  slave 配置
 
  localhost
  server1
  server2
  server3
 
  ----------------------------------------------------------------------------------------------------------------------
 
6.拷贝
  将配置好的hadoop 文件拷贝到所有的 slave中
 
7.启动
  启动之前 现format下hdfs文件
  bin/hadoop namenode -format
 
  bin/start-all.sh
  bin/stop-all.sh
 
8.测试是否启动成功

  jps 命令 ：显示如下
  suse@localhost:/server/bin/hadoop/logs> jps

  12490 TaskTracker
  11854 NameNode
  12343 JobTracker
  12706 Jps
  3832 SecondaryNameNode
  11992 DataNode
  suse@localhost:/server/

http://localhost:50070 (查看节点数)
http://localhost:50030 (查看任务执行情况)