byx5185
- 浏览: 113208 次
- 性别:
- 来自: 上海
-
最新评论
-
bo_hai：
Assembly 打出来的包没有加时间戳呢？
使用Maven assembly打多个自定义包及War包简介 -
woyun1990：
用这个类，生成log的时候会多出个空文件（名称是xml里par ...
支持设置文件大小的DailyRollingFileAppender -
byx5185：
myxiaoribeng 写道 项目中需要， ...
支持设置文件大小的DailyRollingFileAppender -
myxiaoribeng：
项目中需要，我自己写不来，帮了我大忙了
支持设置文件大小的DailyRollingFileAppender -
hotsmile：
modules是啥？
java RSA 加解密
摘要
[root@localhost classes]# hadoop jar first-hadoop-0.0.1-SNAPSHOT.jar hello.Hello
12/04/16 10:41:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/16 10:41:26 INFO mapred.FileInputFormat: Total input paths to process : 1
12/04/16 10:41:26 INFO mapred.JobClient: Running job: job_201204161037_0001
12/04/16 10:41:27 INFO mapred.JobClient: map 0% reduce 0%
12/04/16 10:41:42 INFO mapred.JobClient: map 100% reduce 0%
12/04/16 10:41:58 INFO mapred.JobClient: map 100% reduce 100%
12/04/16 10:42:03 INFO mapred.JobClient: Job complete: job_201204161037_0001
12/04/16 10:42:03 INFO mapred.JobClient: Counters: 30
12/04/16 10:42:03 INFO mapred.JobClient: Job Counters
12/04/16 10:42:03 INFO mapred.JobClient: Launched reduce tasks=1
12/04/16 10:42:03 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=18628
12/04/16 10:42:03 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/04/16 10:42:03 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/04/16 10:42:03 INFO mapred.JobClient: Launched map tasks=2
12/04/16 10:42:03 INFO mapred.JobClient: Data-local map tasks=2
12/04/16 10:42:03 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=13856
12/04/16 10:42:03 INFO mapred.JobClient: File Input Format Counters
12/04/16 10:42:03 INFO mapred.JobClient: Bytes Read=1996
12/04/16 10:42:03 INFO mapred.JobClient: File Output Format Counters
12/04/16 10:42:03 INFO mapred.JobClient: Bytes Written=506
12/04/16 10:42:03 INFO mapred.JobClient: FileSystemCounters
12/04/16 10:42:03 INFO mapred.JobClient: FILE_BYTES_READ=696
12/04/16 10:42:03 INFO mapred.JobClient: HDFS_BYTES_READ=2166
12/04/16 10:42:03 INFO mapred.JobClient: FILE_BYTES_WRITTEN=64709
12/04/16 10:42:03 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=506
12/04/16 10:42:03 INFO mapred.JobClient: Map-Reduce Framework
12/04/16 10:42:03 INFO mapred.JobClient: Map output materialized bytes=702
12/04/16 10:42:03 INFO mapred.JobClient: Map input records=23
12/04/16 10:42:03 INFO mapred.JobClient: Reduce shuffle bytes=702
12/04/16 10:42:03 INFO mapred.JobClient: Spilled Records=92
12/04/16 10:42:03 INFO mapred.JobClient: Map output bytes=598
12/04/16 10:42:03 INFO mapred.JobClient: Total committed heap usage (bytes)=337780736
12/04/16 10:42:03 INFO mapred.JobClient: CPU time spent (ms)=2040
12/04/16 10:42:03 INFO mapred.JobClient: Map input bytes=1330
12/04/16 10:42:03 INFO mapred.JobClient: SPLIT_RAW_BYTES=170
12/04/16 10:42:03 INFO mapred.JobClient: Combine input records=0
12/04/16 10:42:03 INFO mapred.JobClient: Reduce input records=46
12/04/16 10:42:03 INFO mapred.JobClient: Reduce input groups=2
12/04/16 10:42:03 INFO mapred.JobClient: Combine output records=0
12/04/16 10:42:03 INFO mapred.JobClient: Physical memory (bytes) snapshot=324034560
12/04/16 10:42:03 INFO mapred.JobClient: Reduce output records=46
12/04/16 10:42:03 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1122484224
12/04/16 10:42:03 INFO mapred.JobClient: Map output records=46
[root@localhost classes]#
Job日志
/var/log/hadoop/root/hadoop-root-jobtracker-localhost.log
2012-04-16 10:41:26,685 INFO org.apache.hadoop.mapred.JobInProgress: job_201204161037_0001: nMaps=2 nReduces=1 max=-1
2012-04-16 10:41:26,688 INFO org.apache.hadoop.mapred.JobTracker: Job job_201204161037_0001 added successfully for user 'root' to queue 'default'
2012-04-16 10:41:26,689 INFO org.apache.hadoop.mapred.AuditLogger: USER=root IP=127.0.0.1 OPERATION=SUBMIT_JOB TARGET=job_201204161037_0001 RESULT=SUCCESS
2012-04-16 10:41:26,690 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201204161037_0001
2012-04-16 10:41:26,690 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201204161037_0001
2012-04-16 10:41:26,834 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /tmp/hadoop-root/mapred/system/job_201204161037_0001/jobToken
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201204161037_0001 = 1330. Number of splits = 2
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201204161037_0001_m_000000 has split on node:/default-rack/localhost
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201204161037_0001_m_000001 has split on node:/default-rack/localhost
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: job_201204161037_0001 LOCALITY_WAIT_FACTOR=1.0
2012-04-16 10:41:26,856 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201204161037_0001 initialized successfully with 2 map tasks and 1 reduce tasks.
2012-04-16 10:41:29,648 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201204161037_0001_m_000003_0' to tip task_201204161037_0001_m_000003, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,703 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000003_0' has completed task_201204161037_0001_m_000003 successfully.
2012-04-16 10:41:35,713 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201204161037_0001_m_000000_0' to tip task_201204161037_0001_m_000000, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201204161037_0001_m_000000
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201204161037_0001_m_000001_0' to tip task_201204161037_0001_m_000001, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201204161037_0001_m_000001
2012-04-16 10:41:41,733 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000000_0' has completed task_201204161037_0001_m_000000 successfully.
2012-04-16 10:41:41,736 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000001_0' has completed task_201204161037_0001_m_000001 successfully.
2012-04-16 10:41:41,741 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201204161037_0001_r_000000_0' to tip task_201204161037_0001_r_000000, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:56,781 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_r_000000_0' has completed task_201204161037_0001_r_000000 successfully.
2012-04-16 10:41:56,790 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201204161037_0001_m_000002_0' to tip task_201204161037_0001_m_000002, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:42:02,797 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000002_0' has completed task_201204161037_0001_m_000002 successfully.
2012-04-16 10:42:02,798 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201204161037_0001 has completed successfully.
2012-04-16 10:42:02,853 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE subfolder at file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,853 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/root/history/job_201204161037_0001_1334544086637_root_Max+temperature to file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,857 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/root/history/job_201204161037_0001_conf.xml to file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000000_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000001_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000002_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000003_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_r_000000_0'
Task日志
/var/log/hadoop/root/hadoop-root-tasktracker-localhost.log
2012-04-16 10:41:29,682 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000003_0 task's state:UNASSIGNED
2012-04-16 10:41:29,688 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000003_0 which needs 1 slots
2012-04-16 10:41:29,688 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000003_0 which needs 1 slots
2012-04-16 10:41:29,822 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user root on this TT.
2012-04-16 10:41:29,902 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_1468721501
2012-04-16 10:41:29,903 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_1468721501 spawned.
2012-04-16 10:41:29,908 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000003_0/taskjvm.sh
2012-04-16 10:41:30,643 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_1468721501 given task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:34,068 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000003_0 0.0% setup
2012-04-16 10:41:34,070 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000003_0 is done.
2012-04-16 10:41:34,070 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000003_0 was -1
2012-04-16 10:41:34,071 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:34,126 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_1468721501 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:34,178 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2012-04-16 10:41:34,178 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName root for UID 0 from the native implementation
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000000_0 task's state:UNASSIGNED
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000001_0 task's state:UNASSIGNED
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000000_0 which needs 1 slots
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000000_0 which needs 1 slots
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201204161037_0001_m_000003_0 not found in cache
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000001_0 which needs 1 slots
2012-04-16 10:41:35,720 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201204161037_0001_m_000001_0 which needs 1 slots
2012-04-16 10:41:35,727 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_-2010691913
2012-04-16 10:41:35,727 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_-2010691913 spawned.
2012-04-16 10:41:35,728 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_843978914
2012-04-16 10:41:35,729 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_843978914 spawned.
2012-04-16 10:41:35,730 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000001_0/taskjvm.sh
2012-04-16 10:41:35,732 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000000_0/taskjvm.sh
2012-04-16 10:41:37,059 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_-2010691913 given task: attempt_201204161037_0001_m_000000_0
2012-04-16 10:41:37,133 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_843978914 given task: attempt_201204161037_0001_m_000001_0
2012-04-16 10:41:40,815 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000000_0 1.0% hdfs://localhost:9000/user/a.txt:0+665
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000000_0 is done.
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000000_0 was 336
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1
2012-04-16 10:41:40,899 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_-2010691913 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:40,907 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000001_0 1.0% hdfs://localhost:9000/user/a.txt:665+665
2012-04-16 10:41:40,908 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000001_0 is done.
2012-04-16 10:41:40,908 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000001_0 was 366
2012-04-16 10:41:40,909 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:40,985 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_843978914 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_r_000000_0 task's state:UNASSIGNED
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_r_000000_0 which needs 1 slots
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_r_000000_0 which needs 1 slots
2012-04-16 10:41:41,763 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_r_1468721501
2012-04-16 10:41:41,763 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_r_1468721501 spawned.
2012-04-16 10:41:41,765 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_r_000000_0/taskjvm.sh
2012-04-16 10:41:43,008 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_r_1468721501 given task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:49,586 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:49,972 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50600, bytes: 336, op: MAPRED_SHUFFLE, cliID: attempt_201204161037_0001_m_000000_0, duration: 20894000
2012-04-16 10:41:49,979 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50601, bytes: 366, op: MAPRED_SHUFFLE, cliID: attempt_201204161037_0001_m_000001_0, duration: 1496000
2012-04-16 10:41:50,639 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:50,654 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:50,800 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_r_000000_0 is in commit-pending, task state:COMMIT_PENDING
2012-04-16 10:41:50,800 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:53,772 INFO org.apache.hadoop.mapred.TaskTracker: Received commit task action for attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:55,609 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 1.0% reduce > reduce
2012-04-16 10:41:55,611 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 1.0% reduce > reduce
2012-04-16 10:41:55,612 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_r_000000_0 is done.
2012-04-16 10:41:55,612 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_r_000000_0 was -1
2012-04-16 10:41:55,617 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:55,656 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_r_1468721501 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000002_0 task's state:UNASSIGNED
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000002_0 which needs 1 slots
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000002_0 which needs 1 slots
2012-04-16 10:41:56,792 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:56,792 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:56,797 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_-919394950
2012-04-16 10:41:56,797 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_-919394950 spawned.
2012-04-16 10:41:56,799 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000002_0/taskjvm.sh
2012-04-16 10:41:57,534 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_-919394950 given task: attempt_201204161037_0001_m_000002_0
2012-04-16 10:41:57,992 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000002_0 0.0%
2012-04-16 10:42:00,964 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000002_0 0.0% cleanup
2012-04-16 10:42:00,965 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000002_0 is done.
2012-04-16 10:42:00,966 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000002_0 was -1
2012-04-16 10:42:00,966 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:42:01,023 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_-919394950 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:42:02,862 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201204161037_0001
2012-04-16 10:42:02,864 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201204161037_0001_m_000002_0 not found in cache
2012-04-16 10:42:02,872 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201204161037_0001 for user-log deletion with retainTimeStamp:1334630522867
Job用户日志目录
/tmp/hadoop-root/mapred/local/userlogs/job_201204161037_0001/attempt_201204161037_0001_m_000000_0/stdout
/tmp/hadoop-root/mapred/local/userlogs/job_201204161037_0001/attempt_201204161037_0001_r_000000_0/stdout
[root@localhost classes]# hadoop jar first-hadoop-0.0.1-SNAPSHOT.jar hello.Hello
12/04/16 10:41:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/16 10:41:26 INFO mapred.FileInputFormat: Total input paths to process : 1
12/04/16 10:41:26 INFO mapred.JobClient: Running job: job_201204161037_0001
12/04/16 10:41:27 INFO mapred.JobClient: map 0% reduce 0%
12/04/16 10:41:42 INFO mapred.JobClient: map 100% reduce 0%
12/04/16 10:41:58 INFO mapred.JobClient: map 100% reduce 100%
12/04/16 10:42:03 INFO mapred.JobClient: Job complete: job_201204161037_0001
12/04/16 10:42:03 INFO mapred.JobClient: Counters: 30
12/04/16 10:42:03 INFO mapred.JobClient: Job Counters
12/04/16 10:42:03 INFO mapred.JobClient: Launched reduce tasks=1
12/04/16 10:42:03 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=18628
12/04/16 10:42:03 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/04/16 10:42:03 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/04/16 10:42:03 INFO mapred.JobClient: Launched map tasks=2
12/04/16 10:42:03 INFO mapred.JobClient: Data-local map tasks=2
12/04/16 10:42:03 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=13856
12/04/16 10:42:03 INFO mapred.JobClient: File Input Format Counters
12/04/16 10:42:03 INFO mapred.JobClient: Bytes Read=1996
12/04/16 10:42:03 INFO mapred.JobClient: File Output Format Counters
12/04/16 10:42:03 INFO mapred.JobClient: Bytes Written=506
12/04/16 10:42:03 INFO mapred.JobClient: FileSystemCounters
12/04/16 10:42:03 INFO mapred.JobClient: FILE_BYTES_READ=696
12/04/16 10:42:03 INFO mapred.JobClient: HDFS_BYTES_READ=2166
12/04/16 10:42:03 INFO mapred.JobClient: FILE_BYTES_WRITTEN=64709
12/04/16 10:42:03 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=506
12/04/16 10:42:03 INFO mapred.JobClient: Map-Reduce Framework
12/04/16 10:42:03 INFO mapred.JobClient: Map output materialized bytes=702
12/04/16 10:42:03 INFO mapred.JobClient: Map input records=23
12/04/16 10:42:03 INFO mapred.JobClient: Reduce shuffle bytes=702
12/04/16 10:42:03 INFO mapred.JobClient: Spilled Records=92
12/04/16 10:42:03 INFO mapred.JobClient: Map output bytes=598
12/04/16 10:42:03 INFO mapred.JobClient: Total committed heap usage (bytes)=337780736
12/04/16 10:42:03 INFO mapred.JobClient: CPU time spent (ms)=2040
12/04/16 10:42:03 INFO mapred.JobClient: Map input bytes=1330
12/04/16 10:42:03 INFO mapred.JobClient: SPLIT_RAW_BYTES=170
12/04/16 10:42:03 INFO mapred.JobClient: Combine input records=0
12/04/16 10:42:03 INFO mapred.JobClient: Reduce input records=46
12/04/16 10:42:03 INFO mapred.JobClient: Reduce input groups=2
12/04/16 10:42:03 INFO mapred.JobClient: Combine output records=0
12/04/16 10:42:03 INFO mapred.JobClient: Physical memory (bytes) snapshot=324034560
12/04/16 10:42:03 INFO mapred.JobClient: Reduce output records=46
12/04/16 10:42:03 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1122484224
12/04/16 10:42:03 INFO mapred.JobClient: Map output records=46
[root@localhost classes]#
Job日志
/var/log/hadoop/root/hadoop-root-jobtracker-localhost.log
2012-04-16 10:41:26,685 INFO org.apache.hadoop.mapred.JobInProgress: job_201204161037_0001: nMaps=2 nReduces=1 max=-1
2012-04-16 10:41:26,688 INFO org.apache.hadoop.mapred.JobTracker: Job job_201204161037_0001 added successfully for user 'root' to queue 'default'
2012-04-16 10:41:26,689 INFO org.apache.hadoop.mapred.AuditLogger: USER=root IP=127.0.0.1 OPERATION=SUBMIT_JOB TARGET=job_201204161037_0001 RESULT=SUCCESS
2012-04-16 10:41:26,690 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201204161037_0001
2012-04-16 10:41:26,690 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201204161037_0001
2012-04-16 10:41:26,834 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /tmp/hadoop-root/mapred/system/job_201204161037_0001/jobToken
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201204161037_0001 = 1330. Number of splits = 2
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201204161037_0001_m_000000 has split on node:/default-rack/localhost
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201204161037_0001_m_000001 has split on node:/default-rack/localhost
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: job_201204161037_0001 LOCALITY_WAIT_FACTOR=1.0
2012-04-16 10:41:26,856 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201204161037_0001 initialized successfully with 2 map tasks and 1 reduce tasks.
2012-04-16 10:41:29,648 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201204161037_0001_m_000003_0' to tip task_201204161037_0001_m_000003, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,703 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000003_0' has completed task_201204161037_0001_m_000003 successfully.
2012-04-16 10:41:35,713 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201204161037_0001_m_000000_0' to tip task_201204161037_0001_m_000000, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201204161037_0001_m_000000
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201204161037_0001_m_000001_0' to tip task_201204161037_0001_m_000001, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201204161037_0001_m_000001
2012-04-16 10:41:41,733 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000000_0' has completed task_201204161037_0001_m_000000 successfully.
2012-04-16 10:41:41,736 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000001_0' has completed task_201204161037_0001_m_000001 successfully.
2012-04-16 10:41:41,741 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201204161037_0001_r_000000_0' to tip task_201204161037_0001_r_000000, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:56,781 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_r_000000_0' has completed task_201204161037_0001_r_000000 successfully.
2012-04-16 10:41:56,790 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201204161037_0001_m_000002_0' to tip task_201204161037_0001_m_000002, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:42:02,797 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000002_0' has completed task_201204161037_0001_m_000002 successfully.
2012-04-16 10:42:02,798 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201204161037_0001 has completed successfully.
2012-04-16 10:42:02,853 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE subfolder at file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,853 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/root/history/job_201204161037_0001_1334544086637_root_Max+temperature to file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,857 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/root/history/job_201204161037_0001_conf.xml to file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000000_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000001_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000002_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000003_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_r_000000_0'
Task日志
/var/log/hadoop/root/hadoop-root-tasktracker-localhost.log
2012-04-16 10:41:29,682 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000003_0 task's state:UNASSIGNED
2012-04-16 10:41:29,688 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000003_0 which needs 1 slots
2012-04-16 10:41:29,688 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000003_0 which needs 1 slots
2012-04-16 10:41:29,822 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user root on this TT.
2012-04-16 10:41:29,902 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_1468721501
2012-04-16 10:41:29,903 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_1468721501 spawned.
2012-04-16 10:41:29,908 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000003_0/taskjvm.sh
2012-04-16 10:41:30,643 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_1468721501 given task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:34,068 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000003_0 0.0% setup
2012-04-16 10:41:34,070 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000003_0 is done.
2012-04-16 10:41:34,070 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000003_0 was -1
2012-04-16 10:41:34,071 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:34,126 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_1468721501 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:34,178 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2012-04-16 10:41:34,178 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName root for UID 0 from the native implementation
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000000_0 task's state:UNASSIGNED
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000001_0 task's state:UNASSIGNED
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000000_0 which needs 1 slots
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000000_0 which needs 1 slots
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201204161037_0001_m_000003_0 not found in cache
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000001_0 which needs 1 slots
2012-04-16 10:41:35,720 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201204161037_0001_m_000001_0 which needs 1 slots
2012-04-16 10:41:35,727 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_-2010691913
2012-04-16 10:41:35,727 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_-2010691913 spawned.
2012-04-16 10:41:35,728 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_843978914
2012-04-16 10:41:35,729 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_843978914 spawned.
2012-04-16 10:41:35,730 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000001_0/taskjvm.sh
2012-04-16 10:41:35,732 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000000_0/taskjvm.sh
2012-04-16 10:41:37,059 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_-2010691913 given task: attempt_201204161037_0001_m_000000_0
2012-04-16 10:41:37,133 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_843978914 given task: attempt_201204161037_0001_m_000001_0
2012-04-16 10:41:40,815 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000000_0 1.0% hdfs://localhost:9000/user/a.txt:0+665
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000000_0 is done.
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000000_0 was 336
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1
2012-04-16 10:41:40,899 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_-2010691913 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:40,907 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000001_0 1.0% hdfs://localhost:9000/user/a.txt:665+665
2012-04-16 10:41:40,908 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000001_0 is done.
2012-04-16 10:41:40,908 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000001_0 was 366
2012-04-16 10:41:40,909 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:40,985 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_843978914 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_r_000000_0 task's state:UNASSIGNED
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_r_000000_0 which needs 1 slots
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_r_000000_0 which needs 1 slots
2012-04-16 10:41:41,763 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_r_1468721501
2012-04-16 10:41:41,763 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_r_1468721501 spawned.
2012-04-16 10:41:41,765 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_r_000000_0/taskjvm.sh
2012-04-16 10:41:43,008 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_r_1468721501 given task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:49,586 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:49,972 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50600, bytes: 336, op: MAPRED_SHUFFLE, cliID: attempt_201204161037_0001_m_000000_0, duration: 20894000
2012-04-16 10:41:49,979 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50601, bytes: 366, op: MAPRED_SHUFFLE, cliID: attempt_201204161037_0001_m_000001_0, duration: 1496000
2012-04-16 10:41:50,639 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:50,654 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:50,800 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_r_000000_0 is in commit-pending, task state:COMMIT_PENDING
2012-04-16 10:41:50,800 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:53,772 INFO org.apache.hadoop.mapred.TaskTracker: Received commit task action for attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:55,609 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 1.0% reduce > reduce
2012-04-16 10:41:55,611 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 1.0% reduce > reduce
2012-04-16 10:41:55,612 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_r_000000_0 is done.
2012-04-16 10:41:55,612 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_r_000000_0 was -1
2012-04-16 10:41:55,617 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:55,656 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_r_1468721501 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000002_0 task's state:UNASSIGNED
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000002_0 which needs 1 slots
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000002_0 which needs 1 slots
2012-04-16 10:41:56,792 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:56,792 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:56,797 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_-919394950
2012-04-16 10:41:56,797 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_-919394950 spawned.
2012-04-16 10:41:56,799 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000002_0/taskjvm.sh
2012-04-16 10:41:57,534 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_-919394950 given task: attempt_201204161037_0001_m_000002_0
2012-04-16 10:41:57,992 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000002_0 0.0%
2012-04-16 10:42:00,964 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000002_0 0.0% cleanup
2012-04-16 10:42:00,965 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000002_0 is done.
2012-04-16 10:42:00,966 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000002_0 was -1
2012-04-16 10:42:00,966 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:42:01,023 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_-919394950 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:42:02,862 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201204161037_0001
2012-04-16 10:42:02,864 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201204161037_0001_m_000002_0 not found in cache
2012-04-16 10:42:02,872 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201204161037_0001 for user-log deletion with retainTimeStamp:1334630522867
Job用户日志目录
/tmp/hadoop-root/mapred/local/userlogs/job_201204161037_0001/attempt_201204161037_0001_m_000000_0/stdout
/tmp/hadoop-root/mapred/local/userlogs/job_201204161037_0001/attempt_201204161037_0001_r_000000_0/stdout
分享到：
发表评论
相关推荐
017 查看Hadoop 日志以及日志的格式和命名组成 018 Hadoop 守护进程服务三种启动停止方式 019 测试环境(HDFS Shell基本命令和运行WordCount程序) 020 结合WordCount实例讲解Hadoop的数据存储和数据计算 021 Hadoop ...
1. JobTracker 是一个 master 服务，负责调度 Job 的每一个子任务 task 运行于 TaskTracker 上，并监控它们。 2. TaskTracker 是运行在多个节点上的 slaver 服务，负责执行具体的 MapReduce 任务。 3. MapReduce 中...
4. **使用可视化工具**：例如 Hadoop JobHistory Server，它可以显示作业的进度、失败任务的原因以及每个任务的执行时间等信息。 5. **自定义错误处理**：在 Mapper 和 Reducer 类中添加异常处理代码，以便捕获并...
3. MapReduce原理与编程模型：深入解析Map函数和Reduce函数的实现，以及如何编写Mapper和Reducer类，理解job、task、input/output format等概念。 4. Hadoop实战：通过实际案例分析，演示如何使用Hadoop解决实际问题...
4. **hadoop.logfile.size**: 日志文件的最大大小，这里设置为10MB。 5. **fs.default.name**: 指定默认的文件系统，如`file:///`表示本地文件系统，`hdfs:///`表示HDFS。 6. **fs.trash.interval**: 设定文件回收...
- hadoop.logfile.size设定日志文件的最大大小为10M。 - hadoop.logfile.count设置日志文件数量为10。 - io.file.buffer.size用于设定流文件的缓冲区大小，通常为4K或8K。 - io.bytes.per.checksum用于设置校验...
例如，`hadoop.logfile.size`定义了日志文件的最大大小，而`io.file.buffer.size`设置了读写操作的缓冲区大小，影响I/O性能。`io.compression.codecs`列出了支持的压缩算法，如DefaultCodec、GzipCodec等。 了解和...
其中，Partitioner 是 Hadoop MapReduce 框架中的关键组件，它负责决定 map 函数产生的中间键值对（key-value pairs）应被哪些 reduce task 处理。自定义 Partitioner 允许用户根据业务需求定制键的分发策略，从而...
TaskTracker作为工作节点，负责接收和执行JobTracker分配的任务，包括MapTask和ReduceTask。JvmManager则管理TaskTracker上的JVM实例，以优化资源利用率。 理解Hadoop源码有助于我们更深入地了解其内部工作机制，...
- 在job提交过程中，FileInputFormat会根据输入文件大小和配置参数决定数据的切分方式，生成多个split，每个split对应一个MapTask。 - 切片的大小会影响MapTask的数量，进而影响整个作业的性能。 以上内容涵盖了...
MapReduce工作流程包括JobTracker（在Hadoop 2.x中被ResourceManager替代）、TaskTracker（被NodeManager替代）以及MapTask和ReduceTask。JobTracker负责任务调度，TaskTracker执行实际的任务。 **1.5 WordCount...
执行完此命令，Hadoop的各个组件如Name Node、Data Node、Job Tracker和Task Tracker将自动启动，你可以通过浏览器访问http://localhost:50030、50060和50070来监控Namenode、TaskTracker和JobTracker的状态。...
- 异常1：`[communicationthread]org.apache.hadoop.mapred.Task:Communicationexception`。这是一种状态报告机制，当出现异常时会触发重试机制（默认3次）。如果没有再次出现异常，则说明问题已解决，无需特别关注...
2. **HADOOP_OPTS**: 这个变量包含了一组特定于Hadoop的Java选项，可能包括安全、日志、调试等相关的配置。在实际运行时，这些选项会被合并到JVM的启动参数中。 3. **CLASSPATH**: 这是Java应用程序寻找类文件的...
- **Task Tracker Web界面**：查看任务执行情况，包括每个节点的任务分配和完成状态。 - **Name Node Web界面**：显示HDFS的文件系统视图，包括文件和目录的元数据。 8. **后续步骤** - 完成本教程后，你可以...
例如，通过设置`mapred.job.reuse.jvm.num.tasks`属性，可以在同一个JVM中重用Map Task，以减少JVM的启动开销。此外，MultiFileInputSplit可以让一个Map Task处理多个输入split，进一步提高效率。 小文件产生的原因...
MapReduce是大数据处理的核心技术之一，特别是在Hadoop生态系统中，它为处理海量数据提供了高效的...在实际应用中，MapReduce常用于日志分析、推荐系统、搜索引擎索引构建等场景，是大数据领域不可或缺的技术之一。