MAC下hadoop开发环境搭建系列（二）

 

Hadoop的安装，hadoop有三种安装方式：

1）Standalone 单机模式，仅用于调试

2）Pseudo-Distributed 伪分布模式，单节点上同时启动NameNode、DataNode、ResourceManager、NodeManager、Secondary Namenode等进程，模拟分布式运行的各个节点。

3）Fully-Distributed 完全分布式模式，真正的hadoop集群，由各司其职的多个节点组成。

 

HADOOP 单机模式安装

 

跟生产一致，hadoop版本选的是cloudera5.0.0版本，下载地址：http://archive.cloudera.com/cdh5/cdh/5/

＃解压安装包

tar zxvf hadoop-2.3.0-cdh5.0.0.tar

＃将hadoop拷贝到安装路径

cp -r hadoop-2.3.0-cdh5.0.0  ~/hadoop

#设置环境变量

cd ~/hadoop

vi  etc/hadoop/hadoop-env.sh

＃增加以下配置

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home

export HADOOP_PREFIX=~/hadoop

#使配置生效

source etc/hadoop/hadoop-env.sh

#这样单机模式就安装好了,验证安装结果是否正确

 

MacBook-Pro:hadoop Administrator$ bin/hadoop version

Hadoop 2.3.0-cdh5.0.0

Subversion git://github.sf.cloudera.com/CDH/cdh.git -r 8e266e052e423af592871e2dfe09d54c03f6a0e8

Compiled by jenkins on 2014-03-28T04:29Z

Compiled with protoc 2.5.0

From source with checksum fae92214f92a3313887764456097e0

This command was run using /Users/Administrator/hadoop/share/hadoop/common/hadoop-common-2.3.0-cdh5.0.0.jar

也可以试着运行案例程序：

MacBook-Pro:hadoop Administrator$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.0.jar pi 10 100

Number of Maps  = 10

Samples per Map = 100

2015-12-17 11:29:44.665 java[1301:55605] Unable to load realm info from SCDynamicStore

2015-12-17 11:30:29,406 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Wrote input for Map #0

Wrote input for Map #1

Wrote input for Map #2

Wrote input for Map #3

Wrote input for Map #4

Wrote input for Map #5

Wrote input for Map #6

Wrote input for Map #7

Wrote input for Map #8

Wrote input for Map #9

Starting Job

2015-12-17 11:30:29,768 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - session.id is deprecated. Instead, use dfs.metrics.session-id

.......

2015-12-17 11:30:31,483 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1365)) - Job job_local671517986_0001 running in uber mode : false

2015-12-17 11:30:31,486 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1372)) -  map 100% reduce 100%

2015-12-17 11:30:31,488 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1383)) - Job job_local671517986_0001 completed successfully

2015-12-17 11:30:31,517 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1390)) - Counters: 30

File System Counters

FILE: Number of bytes read=3063418

FILE: Number of bytes written=5427585

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

Map-Reduce Framework

Map input records=10

Map output records=20

Map output bytes=180

Map output materialized bytes=280

Input split bytes=1470

Combine input records=0

Combine output records=0

Reduce input groups=2

Reduce shuffle bytes=280

Reduce input records=20

Reduce output records=0

Spilled Records=40

Shuffled Maps =10

Failed Shuffles=0

Merged Map outputs=10

GC time elapsed (ms)=99

Total committed heap usage (bytes)=4572839936

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=1300

File Output Format Counters

Bytes Written=109

Job Finished in 1.778 seconds

Estimated value of Pi is 3.14800000000000000000

 

纪录实际操作过程                                 

内容在个人公众号mangrendd同步更新