Hadoop学习 之单节点集群配置

Hadoop学习 之单节点集群配置

1.官网下载Hadoop
    http://hadoop.apache.org/releases.html 下载hadoop-2.7.3.tar.gz 在hadoop工作目录 eg:  ~/SoftWare/BigData/Hadoop
2 cd eg:  ~/SoftWare/BigData/Hadoop/
   tar -zxvf adoop-2.7.3.tar.gz
3.检查JDK是否安装，如未安装，则安装JDK7+，并设定$JAVA_HOME, $PATH, $CLASSPATH
4 检查ssh, rsync是否安装
    若未安装，则安装
          $ sudo apt-get install ssh
          $ sudo apt-get install rsync
5 修改${HADOOP_HOME}/etc/hadoop/hadoop-env.sh中JAVA_HOME设置，使用实际的绝对路径
  eg:export JAVA_HOME=/home/username/SoftWare/Java/jdk1.8.0_65
6 Standalone Operation下执行一个mapreduce计算
  $ mkdir input
  $ cp etc/hadoop/*.xml input
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'
  $ cat output/*
  可以看到
  $dfsadmin   1

此时表明Standalone Operation下，Hadoop运行成功

7 Pseudo-Distributed Operation
  1)修改${HADOOP_HOME}/etc/hadoop/core-site.xml
     <configuration>
      <property>
          <!-- 指定HDFS老大（namenode）的通信地址 -->
          <name>fs.defaultFS</name>    
          <value>hdfs://localhost:9000</value>
      </property>
      <property>
          <!-- 指定hadoop运行时产生文件的存储路径 -->
          <name>hadoop.tmp.dir</name>
          <value>/home/username/SoftWare/BigData/Hadoop/tmp</value>
      </property>
    </configuration>
  2)修改${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
     <configuration>
       <property>
        <!-- 设置hdfs副本数量 -->
        <name>dfs.replication</name>
        <value>1</value>
      </property>
    </configuration>
  3）检查SSH可否无密码登陆localhost
     $ ssh localhost
  4) 若无需密码SSH可以登陆localhost,则忽略此步，若需要密码，则执行
     $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
     $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
     $ chmod 0600 ~/.ssh/authorized_keys
  5）HDFS的启动与执行
     HDFS的初次执行需要先格式化，执行${HADOOP_HOME}/bin/hdfs namenode -format
     启动HDFS  ${HADOOP_HOME}/sbin/start-dfs.sh
     使用JPS查看HDFS是否启动成功
     $ jps
    
      63842 SecondaryNameNode
      63381 NameNode
      8470 Jps
      63565 DataNode
     看到NameNode,DataNode,SecondaryNameNode进程已经启动，表示HDFS启动成功
     此时可通过 http://localhost:50070/ 查看namenode, 通过 http://localhost:50090 查看datanode
    6) 上传并测试一个文件
         本地创建一个文件words.txt， 内容如下
          Hello World!
          Hello China!
          Hello Jim
         Hello Tom
         The People's Republic Of China!
         上传words.txt至HDFS根目录 ${HADOOP_HOME}/bin/hadoop fs -put words.txt  /
         此时可在http://localhost:50070/explorer.html#/  查看到上传的文件
    7）运行一个例子$ ${HADOOP_HOME}/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'
    查看结果$  ${HADOOP_HOME}/bin/hdfs dfs -cat output/*
        
China!  2
Hello   4
Jim     1
Of      1
People's        1
Republic        1
The     1
Tom     1
World!  1

   8)Pseudo-Distributed Operation下mapreduce可以运行在Yarn上，若需运行在Yarn上
     则修改${HADOOP_HOME}/etc/hadoop/mapred-site.xml
 <configuration>
        <property>
           <!-- 通知框架MR使用YARN -->
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
     </configuration>
     修改${HADOOP_HOME}/etc/hadoop/yarn-site.xml

      <configuration>

<!-- Site specific YARN configuration properties -->
 <property>
            <!-- reducer取数据的方式是mapreduce_shuffle -->
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
 </property>
 <property>
            <!--表示该节点上YARN可使用的物理内存总量，默认是8192（MB），注意，如果你的节点内存资源不够8GB，则需要调减小这个值，而YARN不会智能的探测节点的物理内存总量。  MB为单位-->
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>3072</value>
 </property>
 <property>
            <!--nodemanager可供分配的最小内存  MB为单位-->
  <name>yarn.nodemanager.minmum-allocation-mb</name>
  <value>2048</value>
 </property>
 <property>
            <!--单个任务可申请的最多物理内存量，默认是8192（MB）  MB为单位-->
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>2048</value>
 </property>
 <property>
  <!--用于磁盘空间检查，低于某一值时，会导致mapreduce无法正常运行-->
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
  <value>99</value>
 </property>
</configuration>

    9）启动yarn, ${HADOOP_HOME}/sbin/start_yarn.sh

    10）使用jps检查yarn是否启动成功
        $ jps
       
13761 SecondaryNameNode
13410 NameNode
13923 ResourceManager
16744 Jps
14057 NodeManager
13567 DataNode

此时表明yarn启动成功
可通过http://localhost:8088/cluster查看cluster信息

11）运行一个mapreduce程序
    ${HADOOP_HOME}/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input /output_wordcount
17/05/13 10:38:05 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/13 10:38:06 INFO input.FileInputFormat: Total input paths to process : 1
17/05/13 10:38:06 INFO mapreduce.JobSubmitter: number of splits:1
17/05/13 10:38:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1494642975142_0001
17/05/13 10:38:07 INFO impl.YarnClientImpl: Submitted application application_1494642975142_0001
17/05/13 10:38:07 INFO mapreduce.Job: The url to track the job: http://tizen-HP-Compaq-Pro-6380-MT:8088/proxy/application_1494642975142_0001/
17/05/13 10:38:07 INFO mapreduce.Job: Running job: job_1494642975142_0001
17/05/13 10:38:13 INFO mapreduce.Job: Job job_1494642975142_0001 running in uber mode : false
17/05/13 10:38:13 INFO mapreduce.Job:  map 0% reduce 0%
17/05/13 10:38:18 INFO mapreduce.Job:  map 100% reduce 0%
17/05/13 10:38:23 INFO mapreduce.Job:  map 100% reduce 100%
17/05/13 10:38:24 INFO mapreduce.Job: Job job_1494642975142_0001 completed successfully
17/05/13 10:38:24 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=113
                FILE: Number of bytes written=237983
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=180
                HDFS: Number of bytes written=71
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=2214
                Total time spent by all reduces in occupied slots (ms)=2302
                Total time spent by all map tasks (ms)=2214
                Total time spent by all reduce tasks (ms)=2302
                Total vcore-milliseconds taken by all map tasks=2214
                Total vcore-milliseconds taken by all reduce tasks=2302
                Total megabyte-milliseconds taken by all map tasks=2267136
                Total megabyte-milliseconds taken by all reduce tasks=2357248
        Map-Reduce Framework
                Map input records=5
                Map output records=13
                Map output bytes=130
                Map output materialized bytes=113
                Input split bytes=102
                Combine input records=13
                Combine output records=9
                Reduce input groups=9
                Reduce shuffle bytes=113
                Reduce input records=9
                Reduce output records=9
                Spilled Records=18
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=88
                CPU time spent (ms)=1410
                Physical memory (bytes) snapshot=445538304
                Virtual memory (bytes) snapshot=3855974400
                Total committed heap usage (bytes)=290979840
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=78
        File Output Format Counters
                Bytes Written=71

      表明运行成功  可通过${HADOOP_HOME}/bin/hadoop fs -cat /output_wordcount/* 查看到
     
China!  2
Hello   4
Jim     1
Of      1
People's        1
Republic        1
The     1
Tom     1
World!  1

 

 

 

 

注：使用Yarn时运行一个MapReduce任务 出现
2017-05-13 10:38:07,465 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start
2017-05-13 10:38:07,465 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start

必须在${HADOOP_HOME}/etc/hadoop/yarn-site.xml 添加
 <property>
  <!--用于磁盘空间检查，低于某一值时，会导致mapreduce无法正常运行-->
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
  <value>99</value>
 </property>