Hadoop 完全分布式安装配置学习（hadoop 2.6.5）

艾伦蓝

浏览: 600239 次
性别:
来自: 厦门

最近访客更多访客>>

stephen830

njdccy

lzy8828

WangJiaX

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

Hadoop

一、环境说明
1、机器：一台物理机和一台虚拟机
2、Linux版本：[Spark@S1PA11 ~]$ cat /etc/issue
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
3、JDK: [spark@S1PA11 ~]$ Java -version
Java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
4、集群节点：两个 S1PA11（Master）,S1PA222(Slave)

二、准备工作
1、安装Java jdk前一篇文章撰写了：http://blog.csdn.net/stark_summer/article/details/42391531
2、ssh免密码验证：http://blog.csdn.net/stark_summer/article/details/42393053
3、下载Hadoop版本：http://mirror.bit.edu.cn/apache/hadoop/common/

三、安装Hadoop
这是下载后的hadoop-2.6.5.tar.gz压缩包，
1、解压 tar -xzvf hadoop-2.6.0.tar.gz
2、move到指定目录下：lanwx@master:/usr$ cd hadoop
3、进入hadoop目录
lanwx@master:/usr/hadoop$ ll
total 152
drwxrwxr-x 12 lanwx lanwx 4096 May 7 18:31 ./
drwxr-xr-x 12 root root 4096 May 4 00:29 ../
drwxrwxr-x 2 lanwx lanwx 4096 Oct 2 2016 bin/
drwxrwxr-x 3 lanwx lanwx 4096 Oct 2 2016 etc/
drwxrwxrwx 3 lanwx lanwx 4096 May 5 03:02 hadoop-administrator/
drwxrwxr-x 2 lanwx lanwx 4096 Oct 2 2016 include/
drwxrwxr-x 2 lanwx lanwx 4096 May 7 18:32 input/
drwxrwxr-x 3 lanwx lanwx 4096 Oct 2 2016 lib/
drwxrwxr-x 2 lanwx lanwx 4096 Oct 2 2016 libexec/
-rw-rw-r-- 1 lanwx lanwx 84853 Oct 2 2016 LICENSE.txt
drwxrwxr-x 2 lanwx lanwx 4096 May 7 18:24 logs/
-rw-rw-r-- 1 lanwx lanwx 14978 Oct 2 2016 NOTICE.txt
-rw-rw-r-- 1 lanwx lanwx 1366 Oct 2 2016 README.txt
drwxrwxr-x 2 lanwx lanwx 4096 Oct 2 2016 sbin/
drwxrwxr-x 4 lanwx lanwx 4096 Oct 2 2016 share/

注意：hadoop-administrator 这个目录需要自己创建用于存放hdfs文件
主要涉及的配置文件有8个：都在/hadoop/etc/hadoop文件夹下，可以用gedit命令对其进行编辑。

~/hadoop/etc/hadoop/hadoop-env.sh
~/hadoop/etc/hadoop/yarn-env.sh
~/hadoop/etc/hadoop/master
~/hadoop/etc/hadoop/slaves
~/hadoop/etc/hadoop/core-site.xml
~/hadoop/etc/hadoop/hdfs-site.xml
~/hadoop/etc/hadoop/mapred-site.xml
~/hadoop/etc/hadoop/yarn-site.xml

配置 hadoop-env.sh文件-->修改JAVA_HOME
# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.7.0_25/
export HADOOP_PREFIX=/usr/hadoop/
配置 yarn-env.sh 文件-->>修改JAVA_HOME
# some Java parameters
export JAVA_HOME=/usr/java/jdk1.7.0_25/
配置master文件
master
配置slaves文件-->>增加slave节点
slave03

配置 core-site.xml 文件

<configuration>
    <property>  
        <name>fs.defaultFS</name>  
        <value>hdfs://master:9000</value>   
    </property>  
   
    <property>  
        <name>hadoop.tmp.dir</name>  
        <value>/usr/hadoop/hadoop-administrator</value>    
    </property>  
</configuration>

配置 hdfs-site.xml 文件 注意在datanode中是0.0.0.0不是master

<configuration>
	 <property>
	    <name>dfs.datanode.ipc.address</name>
	    <value>master:50020</value>
  	  </property>
	  <property>
	    <name>dfs.datanode.http.address</name>
	    <value>master:50075</value>
	  </property>
	  <property>
	    <name>dfs.replication</name>
	    <value>1</value>
	  </property>
</configuration>

配置 mapred-site.xml 文件

<configuration>
	 <property>
	    <name>mapreduce.framework.name</name>
	    <value>yarn</value>
	  </property> 
	 <property>
	  <name>mapreduce.jobhistory.address</name>
	  <value>master:10020</value>
	 </property>
	 <property>
	  <name>mapreduce.jobhistory.webapp.address</name>
	  <value>master:19888</value>
	 </property>
</configuration>

配置 yarn-site.xml 文件

<configuration>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>
   <name>yarn.resourcemanager.address</name>
   <value>master:8032</value>
  </property>
  <property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>master:8030</value>
  </property>
  <property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>master:8035</value>
  </property>
  <property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>master:8033</value>
  </property>
  <property>
   <name>yarn.resourcemanager.webapp.address</name>
   <value>master:8088</value>
  </property>

</configuration>

设置环境变量

# set JAVA PATH
export JAVA_HOME=/usr/java/jdk1.7.0_25
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

#hadoop path  
export HADOOP_HOME=/usr/hadoop/ 
export HADOOP_CONF_DIR=/usr/hadoop/etc/hadoop  
export PATH=$HADOOP_HOME/bin:$PATH

将配置好的hadoop文件copy到另一台slave机器上
lanwx@master:/usr$ scp -r hadoop/ slave03:/usr
四、验证
1、格式化namenode:
lanwx@master:/usr$ cd hadoop/
lanwx@master:/usr$ ls
bin dfs etc include input lib libexec LICENSE.txt logs NOTICE.txt README.txt sbin share tmp
lanwx@master:/usr$ ./bin/hdfs namenode -format
lanwx@master:/usr$ cd ~/opt/hadoop-2.6.0
lanwx@master:/usr$ ./bin/hdfs namenode -format
2、启动hadoop:
lanwx@master:/usr/hadoop$ ./sbin/start-all.sh
3、查看hadoop:
jps

9、运行wordcount程序
9.1、创建 input目录：[spark@S1PA11 hadoop-2.6.0]$ mkdir input
9.2、在input创建f1、f2并写内容
[spark@S1PA11 hadoop-2.6.0]$ cat input/f1
Hello world bye jj
[spark@S1PA11 hadoop-2.6.0]$ cat input/f2
Hello Hadoop bye Hadoop
9.3、在hdfs创建/tmp/input目录
[spark@S1PA11 hadoop-2.6.0]$ ./bin/hadoop fs -mkdir /tmp
15/01/05 16:53:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[spark@S1PA11 hadoop-2.6.0]$ ./bin/hadoop fs -mkdir /tmp/input
15/01/05 16:54:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
9.4、将f1、f2文件copy到hdfs /tmp/input目录
[spark@S1PA11 hadoop-2.6.0]$ ./bin/hadoop fs -put input/ /tmp
15/01/05 16:56:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
9.5、查看hdfs上是否有f1、f2文件
[spark@S1PA11 hadoop-2.6.0]$ ./bin/hadoop fs -ls /tmp/input/
15/01/05 16:57:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 3 spark supergroup 20 2015-01-04 19:09 /tmp/input/f1
-rw-r--r-- 3 spark supergroup 25 2015-01-04 19:09 /tmp/input/f2
9.6、执行wordcount程序
[spark@S1PA11 hadoop-2.6.0]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /tmp/input /output
15/01/05 17:00:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/01/05 17:00:09 INFO client.RMProxy: Connecting to ResourceManager at S1PA11/10.58.44.47:8032
15/01/05 17:00:11 INFO input.FileInputFormat: Total input paths to process : 2
15/01/05 17:00:11 INFO mapreduce.JobSubmitter: number of splits:2
15/01/05 17:00:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1420447392452_0001
15/01/05 17:00:12 INFO impl.YarnClientImpl: Submitted application application_1420447392452_0001
15/01/05 17:00:12 INFO mapreduce.Job: The url to track the job: http://S1PA11:8088/proxy/application_1420447392452_0001/
15/01/05 17:00:12 INFO mapreduce.Job: Running job: job_1420447392452_0001
9.7、查看执行结果
[spark@S1PA11 hadoop-2.6.0]$ ./bin/hadoop fs -cat /output/part-r-0000
15/01/05 17:06:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

参考：http://blog.csdn.net/stark_summer/article/details/42424279

查看图片附件

分享到：

Hadoop MapReduce编程模型之InputFormat接 ... | Hadoop 之shuffle（将map输出转化为redu ...

2017-03-15 15:35
浏览 731
评论(0)
分类:编程语言
查看更多

发表评论

文章已被作者锁定，不允许评论。

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论