zhwj184
- 浏览: 755411 次
- 性别:
- 来自: 杭州
-
文章分类
最新评论
-
lgh1992314：
a offset: 26b offset: 24c offse ...
java jvm字节占用空间分析 -
ls0609：
语音实现在线听书http://blog.csdn.net/ls ...
Android 语音输入API使用 -
wangli61289：
http://viralpatel-net-tutorials ...
Android 语音输入API使用 -
zxjlwt：
学习了素人派http://surenpi.com
velocity宏加载顺序 -
tt5753：
谢啦........
Lucene的IndexWriter初始化时的LockObtainFailedException的解决方法
1.先去hadoop官网下载hadoop的源码 http://svn.apache.org/repos/asf/hadoop/common/trunk
2.下载maven3，当前hadoop的最新版必须使用maven3编译
3.到hadoop下载源码目录执行mvn clean install；mvn eclipse:eclipse；
4.将源码导入eclipse；
5.在eclipse设置执行的WordCount.java的jvm启动参数，最少需要两个，输入目录和输出目录
6.然后就可以设置断点进行调试了，我们在处理mapreduce的主干流程上设置断点
org.apache.hadoop.mapred.LocalJobRunner这个类的run方法上
我们看到在我们设置的输入输出目录，然后使用默认的hadoop单机配置下，mapTask有16个，reduceTask有1个
我们先看看我们的输入目录，刚好是16个文件，说明每个输入文件默认启动一个mapTask
而reduce怎么是一个，怎么处理16个mapTask的输出呢
在org.apache.hadoop.mapred.ReduceTask这个reduce处理中run方法中会对所有的map输出做一个merge，然后作为reduceTask的输入
if (!isLocal) {
Class combinerClass = conf.getCombinerClass();
CombineOutputCollector combineCollector =
(null != combinerClass) ?
new CombineOutputCollector(reduceCombineOutputCounter, reporter, conf) : null;
Shuffle shuffle =
new Shuffle(getTaskID(), job, FileSystem.getLocal(job), umbilical,
super.lDirAlloc, reporter, codec,
combinerClass, combineCollector,
spilledRecordsCounter, reduceCombineInputCounter,
shuffledMapsCounter,
reduceShuffleBytes, failedShuffleCounter,
mergedMapOutputsCounter,
taskStatus, copyPhase, sortPhase, this,
mapOutputFile);
rIter = shuffle.run();
} else {
// local job runner doesn't have a copy phase
copyPhase.complete();
final FileSystem rfs = FileSystem.getLocal(job).getRaw();
rIter = Merger.merge(job, rfs, job.getMapOutputKeyClass(),
job.getMapOutputValueClass(), codec,
getMapFiles(rfs, true),
!conf.getKeepFailedTaskFiles(),
job.getInt(JobContext.IO_SORT_FACTOR, 100),
new Path(getTaskID().toString()),
job.getOutputKeyComparator(),
reporter, spilledRecordsCounter, null, null);
}
由于用到的是单机模式，所以没有用到Shuffle的过程。
整个日志输出如下：
2012-11-05 12:46:37,006 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - session.id is deprecated. Instead, use dfs.metrics.session-id 2012-11-05 12:46:37,050 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId= 2012-11-05 12:46:37,195 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2012-11-05 12:46:37,263 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(247)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2012-11-05 12:46:37,315 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 16 2012-11-05 12:46:37,784 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(368)) - number of splits:16 2012-11-05 12:46:37,832 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name 2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 2012-11-05 12:46:37,834 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2012-11-05 12:46:37,834 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 2012-11-05 12:46:37,834 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 2012-11-05 12:46:37,835 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 2012-11-05 12:46:38,048 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(438)) - Submitting tokens for job: job_local_0001 2012-11-05 12:46:38,597 INFO [main] mapreduce.Job (Job.java:submit(1222)) - The url to track the job: http://localhost:8080/ 2012-11-05 12:46:38,663 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1267)) - Running job: job_local_0001 2012-11-05 12:46:38,666 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(320)) - OutputCommitter set in config null 2012-11-05 12:46:38,677 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(338)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2012-11-05 12:46:39,666 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1288)) - Job job_local_0001 running in uber mode : false 2012-11-05 12:46:39,668 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 0% reduce 0% 2012-11-05 12:52:04,731 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000000_0 2012-11-05 12:52:04,894 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@16c4642 2012-11-05 12:52:04,903 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/capacity-scheduler.xml:0+7457 2012-11-05 12:52:05,006 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:05,062 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 10166; bufvoid = 104857600 2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26211052(104844208); length = 3345/6553600 2012-11-05 12:52:05,151 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:05,155 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000000_0 is done. And is in the process of committing 2012-11-05 12:52:05,166 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:05,166 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000000_0' done. 2012-11-05 12:52:05,166 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000000_0 2012-11-05 12:52:05,852 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 100% reduce 0% 2012-11-05 12:52:06,389 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000001_0 2012-11-05 12:52:06,389 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:run(386)) - Waiting for map tasks 2012-11-05 12:52:06,394 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1a3c2bf 2012-11-05 12:52:06,398 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hadoop-policy.xml:0+4644 2012-11-05 12:52:06,466 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:06,466 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:06,466 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:06,467 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:06,467 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:06,483 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:06,483 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:06,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:06,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 6454; bufvoid = 104857600 2012-11-05 12:52:06,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26212228(104848912); length = 2169/6553600 2012-11-05 12:52:06,508 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:06,515 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000001_0 is done. And is in the process of committing 2012-11-05 12:52:06,520 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:06,520 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000001_0' done. 2012-11-05 12:52:06,520 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000001_0 2012-11-05 12:52:06,521 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000002_0 2012-11-05 12:52:06,528 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@adc92c 2012-11-05 12:52:06,529 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/log4j.properties:0+4441 2012-11-05 12:52:06,615 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:06,724 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 5492; bufvoid = 104857600 2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26213316(104853264); length = 1081/6553600 2012-11-05 12:52:06,737 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:06,741 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000002_0 is done. And is in the process of committing 2012-11-05 12:52:06,743 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:06,744 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000002_0' done. 2012-11-05 12:52:06,744 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000002_0 2012-11-05 12:52:06,744 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000003_0 2012-11-05 12:52:06,747 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1c6f86d 2012-11-05 12:52:06,749 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hadoop-env.sh:0+2237 2012-11-05 12:52:07,288 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,288 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,288 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,289 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,289 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 3255; bufvoid = 104857600 2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26213356(104853424); length = 1041/6553600 2012-11-05 12:52:07,330 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:07,333 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000003_0 is done. And is in the process of committing 2012-11-05 12:52:07,334 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:07,335 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000003_0' done. 2012-11-05 12:52:07,335 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000003_0 2012-11-05 12:52:07,335 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000004_0 2012-11-05 12:52:07,339 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@19d34ca 2012-11-05 12:52:07,401 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/mapred-queue-acls.xml:0+2033 2012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:07,484 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:07,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:07,485 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 3020; bufvoid = 104857600 2012-11-05 12:52:07,485 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26213292(104853168); length = 1105/6553600 2012-11-05 12:52:07,494 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:07,496 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000004_0 is done. And is in the process of committing 2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000004_0' done. 2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000004_0 2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000005_0 2012-11-05 12:52:07,500 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1fe3515 2012-11-05 12:52:07,502 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hadoop-metrics2.properties:0+1488 2012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 1710; bufvoid = 104857600 2012-11-05 12:52:07,567 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214160(104856640); length = 237/6553600 2012-11-05 12:52:07,574 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:07,577 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000005_0 is done. And is in the process of committing 2012-11-05 12:52:07,582 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:07,582 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000005_0' done. 2012-11-05 12:52:07,583 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000005_0 2012-11-05 12:52:07,583 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000006_0 2012-11-05 12:52:07,585 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@4bc5a5 2012-11-05 12:52:07,587 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/ssl-client.xml.example:0+1243 2012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,698 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 1530; bufvoid = 104857600 2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214044(104856176); length = 353/6553600 2012-11-05 12:52:07,709 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:07,712 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000006_0 is done. And is in the process of committing 2012-11-05 12:52:07,713 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:07,714 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000006_0' done. 2012-11-05 12:52:07,714 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000006_0 2012-11-05 12:52:07,714 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000007_0 2012-11-05 12:52:07,716 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@15f7c4b 2012-11-05 12:52:07,718 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/ssl-server.xml.example:0+1195 2012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:07,788 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 1470; bufvoid = 104857600 2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214060(104856240); length = 337/6553600 2012-11-05 12:52:07,793 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:07,795 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000007_0 is done. And is in the process of committing 2012-11-05 12:52:07,796 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:07,797 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000007_0' done. 2012-11-05 12:52:07,797 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000007_0 2012-11-05 12:52:07,797 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000008_0 2012-11-05 12:52:07,798 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@17eb194 2012-11-05 12:52:07,799 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/configuration.xsl:0+535 2012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 666; bufvoid = 104857600 2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600 2012-11-05 12:52:07,866 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:07,869 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000008_0 is done. And is in the process of committing 2012-11-05 12:52:07,870 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:07,871 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000008_0' done. 2012-11-05 12:52:07,871 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000008_0 2012-11-05 12:52:07,871 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000009_0 2012-11-05 12:52:07,873 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@14924fb 2012-11-05 12:52:07,874 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/taskcontroller.cfg:0+382 2012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:07,994 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 546; bufvoid = 104857600 2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214236(104856944); length = 161/6553600 2012-11-05 12:52:08,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,010 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000009_0 is done. And is in the process of committing 2012-11-05 12:52:08,011 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,011 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000009_0' done. 2012-11-05 12:52:08,012 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000009_0 2012-11-05 12:52:08,012 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000010_0 2012-11-05 12:52:08,014 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1e54167 2012-11-05 12:52:08,016 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/fair-scheduler.xml:0+327 2012-11-05 12:52:08,075 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 460; bufvoid = 104857600 2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214256(104857024); length = 141/6553600 2012-11-05 12:52:08,088 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,091 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000010_0 is done. And is in the process of committing 2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000010_0' done. 2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000010_0 2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000011_0 2012-11-05 12:52:08,094 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@ccde81 2012-11-05 12:52:08,095 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/core-site.xml:0+178 2012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 239; bufvoid = 104857600 2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600 2012-11-05 12:52:08,162 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,168 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000011_0 is done. And is in the process of committing 2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000011_0' done. 2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000011_0 2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000012_0 2012-11-05 12:52:08,172 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@fbee67 2012-11-05 12:52:08,173 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/mapred-site.xml:0+178 2012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 239; bufvoid = 104857600 2012-11-05 12:52:08,392 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600 2012-11-05 12:52:08,398 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,402 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000012_0 is done. And is in the process of committing 2012-11-05 12:52:08,404 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,405 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000012_0' done. 2012-11-05 12:52:08,405 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000012_0 2012-11-05 12:52:08,405 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000013_0 2012-11-05 12:52:08,407 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@ac8211 2012-11-05 12:52:08,409 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hdfs-site.xml:0+178 2012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,487 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 239; bufvoid = 104857600 2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600 2012-11-05 12:52:08,491 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,493 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000013_0 is done. And is in the process of committing 2012-11-05 12:52:08,495 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,496 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000013_0' done. 2012-11-05 12:52:08,496 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000013_0 2012-11-05 12:52:08,496 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000014_0 2012-11-05 12:52:08,498 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@5a4a14 2012-11-05 12:52:08,500 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/masters:0+10 2012-11-05 12:52:08,558 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 14; bufvoid = 104857600 2012-11-05 12:52:08,563 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600 2012-11-05 12:52:08,565 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,568 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000014_0 is done. And is in the process of committing 2012-11-05 12:52:08,570 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,570 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000014_0' done. 2012-11-05 12:52:08,570 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000014_0 2012-11-05 12:52:08,571 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000015_0 2012-11-05 12:52:08,572 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@527e31 2012-11-05 12:52:08,574 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/slaves:0+10 2012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584) 2012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 100 2012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 83886080 2012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 104857600 2012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 6553600 2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output 2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output 2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 14; bufvoid = 104857600 2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600 2012-11-05 12:52:08,693 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 0 2012-11-05 12:52:08,695 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000015_0 is done. And is in the process of committing 2012-11-05 12:52:08,698 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map 2012-11-05 12:52:08,699 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000015_0' done. 2012-11-05 12:52:08,699 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000015_0 2012-11-05 12:52:08,699 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:run(394)) - Map task executor complete. 2012-11-05 12:52:58,415 INFO [Thread-10] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@e85102 2012-11-05 12:58:16,363 INFO [Thread-10] mapred.Merger (Merger.java:merge(549)) - Merging 16 sorted segments 2012-11-05 12:59:41,406 INFO [Thread-10] mapred.Merger (Merger.java:merge(653)) - Merging 7 intermediate segments out of a total of 16 2012-11-05 12:59:44,605 INFO [Thread-10] mapred.Merger (Merger.java:merge(648)) - Down to the last merge-pass, with 10 segments left of total size: 22378 bytes 2012-11-05 12:59:44,966 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:59:46,437 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce 2012-11-05 12:59:46,486 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 100% reduce 66% 2012-11-05 12:59:47,163 WARN [Thread-10] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2012-11-05 12:59:47,198 INFO [Thread-10] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_r_000000_0 is done. And is in the process of committing 2012-11-05 12:59:47,198 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce 2012-11-05 12:59:47,198 INFO [Thread-10] mapred.Task (Task.java:commit(1141)) - Task attempt_local_0001_r_000000_0 is allowed to commit now 2012-11-05 12:59:47,199 INFO [Thread-10] output.FileOutputCommitter (FileOutputCommitter.java:commitTask(432)) - Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/weijianzhongwj/software/hadoop-1.1.0/out/_temporary/0/task_local_0001_r_000000 2012-11-05 12:59:47,199 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce 2012-11-05 12:59:47,200 INFO [Thread-10] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_r_000000_0' done. 2012-11-05 12:59:47,486 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 100% reduce 100% 2012-11-05 12:59:47,487 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1306)) - Job job_local_0001 completed successfully 2012-11-05 12:59:47,602 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1313)) - Counters: 27 File System Counters FILE: Number of bytes read=667992 FILE: Number of bytes written=2711113 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=749 Map output records=2585 Map output bytes=35514 Map output materialized bytes=22535 Input split bytes=2154 Combine input records=2585 Combine output records=1089 Reduce input groups=789 Reduce shuffle bytes=0 Reduce input records=1089 Reduce output records=789 Spilled Records=2293 Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=603 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=4293918720 File Input Format Counters Bytes Read=26536 File Output Format Counters Bytes Written=15413
有很多的统计信息
我们再看下最终的输出目录：
0
顶
顶
0
踩
踩
分享到：
发表评论
-
对字符串进行验证之前先进行规范化
2013-09-17 23:18 13957对字符串进行验证之前先进行规范化 应用系统中经常对字 ... -
在Apache Tomcat 7设置redis作为session store
2013-08-21 17:00 1994原文：http://www.javaarch.ne ... -
使用telnet连接到基于spring的应用上执行容器中的bean的任意方法
2013-08-08 09:17 1482使用telnet连接到基于spring的应用上执行容器中 ... -
jdk7和8的一些新特性介绍
2013-07-06 16:07 10114更多ppt内容请查看：htt ... -
java对于接口和抽象类的代理实现，不需要有具体实现类
2013-06-12 09:50 2957原文链接：http://www.javaarch.net/j ... -
Excel2007格式分析和XML解析
2013-06-07 09:56 10746在物料清单采购中，用到excel上传文件解析功能，不 ... -
Java EE 7中对WebSocket 1.0的支持
2013-06-05 09:27 3846原文链接：http://www.javaarch.n ... -
java QRCode生成示例
2013-06-05 09:26 1516原文链接：http://www.javaarch.n ... -
Java Web使用swfobject调用flex图表
2013-05-28 19:05 1128Java Web使用swfobject调用 ... -
spring使用PropertyPlaceholderConfigurer扩展来满足不同环境的参数配置
2013-05-21 15:57 3344spring使用PropertyPlaceholderCon ... -
java国际化
2013-05-20 20:57 4478java国际化 本文来自：http://www.j ... -
RSS feeds with Java
2013-05-20 20:52 1226RSS feeds with Java 原文来自：htt ... -
使用ibatis将数据库从oracle迁移到mysql的几个修改点
2013-04-29 10:40 1679我们项目在公司的大战略下需要从oracle ... -
线上机器jvm dump分析脚本
2013-04-19 10:48 2912#!/bin/sh DUMP_PIDS=`p ... -
eclipse远程部署，静态文件实时同步插件
2013-04-06 20:18 5469eclipse 远程文件实时同步，eclipse远程 ... -
java价格处理的一个问题
2013-03-26 21:21 1841我们经常会处理一些价格，比如从运营上传的文件中将某 ... -
java 服务降级开关设计思路
2013-03-23 16:35 3772java 服务屏蔽开关系统，可以手工降级服务，关闭服 ... -
poi解析excel内存溢出
2013-03-20 22:21 6406真是悲剧啊，一个破内部使用系统20多个人使用的后 ... -
简单web安全框架
2013-03-16 11:56 1551web安全框架,主要用servlet filter方 ... -
基于servlet的简单的页面缓存框架
2013-03-11 19:27 1222基于servlet的页面级缓存框架的基本用法： 代码参考： ...
相关推荐
### Ubuntu 下的Hadoop配置与运行 #### 一、系统配置与环境搭建 ...通过以上步骤，可以在 Ubuntu 系统下完成 Hadoop 的基本配置，并实现单节点运行 WordCount 示例。这为后续探索更复杂的分布式计算场景打下了基础。
- 将WordCount示例程序编译成jar包，然后使用Hadoop的`hadoop jar`命令提交到集群执行。 6. **监控和调试**： - 使用Hadoop提供的Web界面监控NameNode和JobTracker的状态。 - 查看日志文件进行故障排查。 通过...
- 可以尝试运行内置的WordCount示例程序来测试Hadoop是否正确配置。 - 进入`/home/shiep205/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-0.20.0.jar`目录。 - 使用命令`bin/hadoop jar hadoop-...
### 在Ubuntu 13.10环境中配置Hadoop ...完成以上步骤后，您就可以在Ubuntu 13.10环境中成功配置并运行Hadoop以及Eclipse上的WordCount示例程序了。这为大数据处理提供了一个稳定且高效的环境基础。
创建新的Java项目，编写MapReduce程序，例如经典的WordCount示例。Map阶段负责切分输入数据并生成键值对，Reduce阶段则对相同键的键值对进行聚合。 ### 查看HDFS文件系统数据的三种方法 1. 使用Hadoop提供的命令行...
**WordCount示例** WordCount是Hadoop的入门示例，用于统计文本中单词出现的次数。主要包含两个部分：Mapper和Reducer。 - **Mapper**：接收输入行，将其按空格分割成单词，然后输出<单词, 1>键值对。 - **...
在这个例子中，操作系统是通过Wubi在Windows上安装的Ubuntu 10.10，Hadoop版本为hadoop-0.20.2，Eclipse版本为eclipse-jee-helios-SR1-linux-gtk。为了简化学习过程，我们将在“伪分布式模式”下运行Hadoop，这意味...
整个安装过程分为六个主要部分：安装虚拟化工具VMware、在VMware上安装Ubuntu系统、安装JDK与SSH服务作为Hadoop安装前的准备、配置Hadoop、安装Eclipse以及运行一个简单的Hadoop程序——WordCount.java。 #### 二、...
6. 测试Wordcount示例：运行Hadoop自带的Wordcount示例，验证环境搭建是否成功。 搭建伪分布式开发环境的步骤大致与单机模式相似，但是需要对Hadoop配置文件进行进一步的配置，以使Hadoop模拟分布式环境运行。 ...
Wordcount 程序是 Hadoop 的一个示例程序，用于统计文本文件中的单词数量。运行 Wordcount 程序可以验证 Hadoop 集群的正确性。 Eclipse 开发环境的建立 Eclipse 是一个流行的集成开发环境（IDE），可以用于开发 ...
WordCount是Hadoop的一个经典示例，用于统计文本中单词出现的次数。Map阶段，每个Map任务将输入行拆分成单词，输出键为单词，值为1。Reduce阶段，所有包含同一单词的值被合并并相加，得到该单词的总出现次数。这展示...
- 从网络下载WordCount示例代码。 - 在MyEclipse的“Run Configurations”中配置输入文件路径和输出文件路径。 3. **运行程序**: - 选择“Run on Hadoop”选项运行程序。 - 成功运行后，将在HDFS的输出目录下...
综上所述，本文介绍了在特定的 Hadoop 和 Spark 集群环境下进行 WordCount 示例的实现过程。从环境搭建、IDE 配置到代码编写，每个步骤都进行了详细的说明。通过学习这个案例，可以帮助读者更好地理解 Spark 的基本...
本文将详细介绍如何在本地环境中搭建Spark开发环境，并通过一个简单的WordCount示例来验证环境是否搭建成功。 #### 相关软件与环境配置 在开始之前，我们需要准备以下软件： - **操作系统**：推荐使用Ubuntu（也...
- **使用Storm开发一个WordCount例子**：通过WordCount示例来演示Storm的应用开发过程。 - **Storm程序本地模式debug、Storm程序远程debug**：指导如何调试Storm程序。 - **Storm事务处理**：介绍Storm如何支持事务...