install hadoop-2.5 without HDFS HA /Federation

 

I. installation mode

  same as hadoop 1.x ,there are several mode to install hadoop:

1.standalone

  just run it on one machine,includes running of mapreduce.

2.pseudo 

 setup it with hdfs mode,and this case contains two types:

 a.run hdfs only 

  in this case,mapreds also run in local mode ,yes ,you can see the job name called as job_localxxxxxx

 b.run hdfs with yarn

  yes ,this is same as the distributed mode

3.distributed mode/cluster mode

  compare to item 2,this item only has some more configures and more than one  nodes. 

 

II.configures for cluster mode

file	property	value	default val	summary
core-site.xml	hadoop.tmp.dir	/usr/local/hadoop/data-2.5.1/tmp	/tmp/hadoop-${user.name}	path to a tmp dir, some sub dirs will be filecache,usercache,nmPrivate.so thisdir shoult not set todir 'tmp' for productenvironment;
	fs.defaultFS	hdfs://host1:9000	file:///	the name of the default file system.this will determine the installation mode ;the correspondent deprecated one is: fs.default.name; The name of the default file system.  A URI whose   scheme and authority determine the FileSystem implementation.  The   uri's scheme determines the config property (fs.SCHEME.impl) naming   the FileSystem implementation class.  The uri's authority is used to     determine the host, port, etc. for a filesystem.
hdfs-site.xml	dfs.nameservices	hadoop-cluster1		Comma-separated list of nameservices.here is single NN only but HA
	dfs.namenode.secondary.http-address	host1:50090	0.0.0.0:50090	The secondary namenode http server address and port.
	dfs.namenode.name.dir	file:///usr/local/hadoop/data-2.5.1/dfs/name	file://${hadoop.tmp.dir}/dfs/name	Determines where on the local filesystem the DFS name node       should store the name table(fsimage).  If this is a comma-delimited list       of directories then the name table is replicated in all of the       directories, for redundancy.
	dfs.datanode.data.dir	file:///usr/local/hadoop/data-2.5.1/dfs/data	file://${hadoop.tmp.dir}/dfs/data	Determines where on the local filesystem an DFS data node   should store its blocks.  If this is a comma-delimited   list of directories, then data will be stored in all named   directories, typically on different devices.   Directories that do not exist are ignored.
	dfs.replication	1	3	the replication factor to assign data blocks
	dfs.webhdfs.enabled	true	true	Enable WebHDFS (REST API) in Namenodes and Datanodes.
yarn-site.xml	yarn.nodemanager.aux-services	mapreduce_shuffle	the auxiliary service name	the valid service name should only contain a-zA-Z0-9_ and can not start with numbers
	yarn.resourcemanager.address	host1:8032	${yarn.resourcemanager.hostname}:8032	The address of the applications manager interface in the RM
	yarn.resourcemanager.scheduler.address	host1:8030	${yarn.resourcemanager.hostname}:8030	the scheduler address of RM
	yarn.resourcemanager.resource-tracker.address	host1:8031	${yarn.resourcemanager.hostname}:8031
	yarn.resourcemanager.admin.address	host1:8033	${yarn.resourcemanager.hostname}:8033	admin addr
	yarn.resourcemanager.webapp.address	host1:50030	${yarn.resourcemanager.hostname}:8088	the webp ui addr for RM ;here is set to job tracker addr that same as hadoop 1.x
mapred-site.xml	mapreduce.framework.name	yarn	local	The runtime framework for executing MapReduce jobs.   Can be one of local, classic or yarn.
	mapreduce.jobhistory.address	host1:10020	0.0.0.0:10020	MapReduce JobHistory Server IPC host:port
	mapreduce.jobhistory.webapp.address	host1:19888	0.0.0.0:19888	MapReduce JobHistory Server Web UI host:port

 

 

 

 

 

 

 

 

III.the results of running MR in yarn

below are logs from mapreduce run with pseudo mode:

hadoop@ubuntu:/usr/local/hadoop/hadoop-2.5.1$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar wordcount wc wc-out

14/11/05 18:19:23 INFO client.RMProxy: Connecting to ResourceManager at namenode/192.168.1.25:8032

14/11/05 18:19:24 INFO input.FileInputFormat: Total input paths to process : 22

14/11/05 18:19:24 INFO mapreduce.JobSubmitter: number of splits:22

14/11/05 18:19:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415182439385_0001

14/11/05 18:19:25 INFO impl.YarnClientImpl: Submitted application application_1415182439385_0001

14/11/05 18:19:25 INFO mapreduce.Job: The url to track the job: http://namenode:50030/proxy/application_1415182439385_0001/

14/11/05 18:19:25 INFO mapreduce.Job: Running job: job_1415182439385_0001

14/11/05 18:19:32 INFO mapreduce.Job: Job job_1415182439385_0001 running in uber mode : false

14/11/05 18:19:32 INFO mapreduce.Job:  map 0% reduce 0%

14/11/05 18:19:44 INFO mapreduce.Job:  map 9% reduce 0%

14/11/05 18:19:45 INFO mapreduce.Job:  map 27% reduce 0%

14/11/05 18:19:54 INFO mapreduce.Job:  map 32% reduce 0%

14/11/05 18:19:55 INFO mapreduce.Job:  map 45% reduce 0%

14/11/05 18:19:56 INFO mapreduce.Job:  map 50% reduce 0%

14/11/05 18:20:02 INFO mapreduce.Job:  map 55% reduce 17%

14/11/05 18:20:03 INFO mapreduce.Job:  map 59% reduce 17%

14/11/05 18:20:05 INFO mapreduce.Job:  map 68% reduce 20%

14/11/05 18:20:06 INFO mapreduce.Job:  map 73% reduce 20%

14/11/05 18:20:08 INFO mapreduce.Job:  map 73% reduce 24%

14/11/05 18:20:11 INFO mapreduce.Job:  map 77% reduce 24%

14/11/05 18:20:12 INFO mapreduce.Job:  map 82% reduce 24%

14/11/05 18:20:13 INFO mapreduce.Job:  map 91% reduce 24%

14/11/05 18:20:14 INFO mapreduce.Job:  map 95% reduce 30%

14/11/05 18:20:16 INFO mapreduce.Job:  map 100% reduce 30%

14/11/05 18:20:17 INFO mapreduce.Job:  map 100% reduce 100%

14/11/05 18:20:18 INFO mapreduce.Job: Job job_1415182439385_0001 completed successfully

14/11/05 18:20:18 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=54637

FILE: Number of bytes written=2338563

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=59677

HDFS: Number of bytes written=28233

HDFS: Number of read operations=69

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters 

Launched map tasks=22

Launched reduce tasks=1

Data-local map tasks=22

Total time spent by all maps in occupied slots (ms)=185554

Total time spent by all reduces in occupied slots (ms)=30206

Total time spent by all map tasks (ms)=185554

Total time spent by all reduce tasks (ms)=30206

Total vcore-seconds taken by all map tasks=185554

Total vcore-seconds taken by all reduce tasks=30206

Total megabyte-seconds taken by all map tasks=190007296

Total megabyte-seconds taken by all reduce tasks=30930944

Map-Reduce Framework

Map input records=1504

Map output records=5727

Map output bytes=77326

Map output materialized bytes=54763

Input split bytes=2498

Combine input records=5727

Combine output records=2838

Reduce input groups=1224

Reduce shuffle bytes=54763

Reduce input records=2838

Reduce output records=1224

Spilled Records=5676

Shuffled Maps =22

Failed Shuffles=0

Merged Map outputs=22

GC time elapsed (ms)=1707

CPU time spent (ms)=14500

Physical memory (bytes) snapshot=5178937344

Virtual memory (bytes) snapshot=22517506048

Total committed heap usage (bytes)=3882549248

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters 

Bytes Read=57179

File Output Format Counters 

Bytes Written=28233

 

FAQs

1.2014-01-22 09:38:20,733 INFO  [AsyncDispatcher event handler] rmapp.RMAppImpl (RMAppImpl.java:transition(788)) - Application application_1390354688375_0001 failed 2 times due to AM Container for appattempt_1390354688375_0001_000002 exited with  exitCode: 127 due to: Exception from container-launch: 

  this maybe occur if you dont setup a JAVA_HOME in yarn-env.sh and hadoop-env.sh,and remember to restart yarn:)

2.occurs two jobs by running 'grep' example

  it's normal!at first ,i think it's some wrong,but when i run wordcount again,the result shows one job only .so i think it's the nature of this example.

 

ref:

apache install hadoop 2