`
sealbird
  • 浏览: 583904 次
  • 性别: Icon_minigender_1
  • 来自: 广州
社区版块
存档分类
最新评论

hadoop job提交完成的整个过程介绍 zz

 
阅读更多
2009-11-17 11:16http://blog.chinaunix.net/u3/94300/showart_1902760.html

由于大量的使用interface,reflection,rpc proxy,所以当我们提交job给hadoop的时候,他到底是如何一步步运行的确实不太容易看明白,今天费了将近一天的功夫终于将其大概整理了出俩,为以后继续深入仔细阅读源码打下基础。

start JobTracker and TaskTracker by bin/start-all.sh
JobTracker initialization:
read all host( master and slave )
init QueueManager to manage job queue
start JobQueueTaskScheduler to schedule job
add JobQueueJobInProcessListener to manage submited jobs
add EagerTaskInitialization, so when job was submited, JobinitThread will get first job in job queue and execute job ( throught job.initTask)
start JobTracker Server which is instance of RPC.Server, derived from org.apache.hadoop.mapred.ipc.Server, so client can submit job by creating proxy
start httpServer (third party library) so we can get status of job tracker through http
init JobTrackerInstrumentation so record measure information
start RecoverManager so we can try to recover unfinished history jobs
init CompletedJobStatusStore to persist and retrive job information
start ExpireTrackers thread to remove all expiry TaskTracker, whose heatbeat response out of time
start RetireJobs to remove finished jobs that have been around for too long ( the job status is not runing, prepare ... )
start ExpireLaunchingTasks to check task assignment failure and reassign the task
join JobTracker Server to wait for job submit request

TaskTracker initialization:
start http server to report map/reduce status by http
start TaskTrackerInstrument to record measure info
init JvmManager which can find free JVM to run task
start taskReportServer (ipc.Server) with instance as TaskTracker
create JobTracker proxy so the taskTracker can communicate with JobTracker
start mapEventsFetcher to fetch all map event
start mapLauncher and reduceLauncher ( instance of TaskLauncher) thread to monitor task queue, if new task was inserted, get first and startNewTask.
after TaskTracker initialization, start TaskTracker server which will looply communicate with JobTracker by Hearbbeat test, also get task.
check buildVersion and file system
purge map-event and reset reducer
check if the TaskTracker should be restart
get TaskAction from Heartbert message, if exist, add action to actionQueue



org.apache.hadoop.mapred.JobClient :
public static RunningJob runJob(JobConf job)

public RunningJob submitJob(JobConf job)
copy job jar to DFS
get input split
sort input split
write split info into file system
jobSubmitClient.submitJob, jobSubmitClient is client proxy of org.apache.hadoop.mapred.JobTracker, which record status of all jobs

org.apache.hadoop.mapred.JobTracker :
public JobStatus submitJob(JobID jobName)
create JobInProcess with jobID
copy job jar from DFS to local
add job to JobQueueJobInProgressListener and EagerTaskInitializationListener, EagerTaskInitializationListener will call JobInProcess.initTask to create map and reduce task based on InputSplit, and create other assistant task, such as clean-up task ect.
JobTracker call JobQueueTaskScheduler.assignTask to assign task for each request host
call JobTracker.getSetupAndCleanupTasks() to get all tasks waited for execution, call TaskInProcess.addRunningTask() to set map or reduce task
all got tasks were encapsulated in TaskActions
send TaskActions to TaskTrackers through HeartbeatReponse

TaskTracker get TaskAction from HeartbeatResponse message ( JobTracker call JobQueueTaskScheduler.assignTask to assign task for each request host)
TaskTracker.TaskInProcess.registerTask() was called so TaskAction was transformed to Tasktracker.TaskInProcess (tip)
the newly got tip was insert into task queue ( named tasksToLaunch) of TaskLauncher ( named mapLauncher and reduceLauncher), once new task was inserted:
TaskTracker.startNewTask was called
TaskTracker.localizeJob() was called
add task to job, so MapEventFetcher can get map event (further thinking)
get job jar file and create necessary working directory
unjar job jar file
TaskTracker.launchTaskForJob was called
TaskTracker.TaskInProcess.launchTask was called
TaskTracker.TaskInProces.localizeTask() was called
create local working dir
create symlink for job dir if it doesn't exist
set resolved hostname
set debug parameter if debugCommand exist
Task.createRunner (MapTask or ReduceTask) was called
TaskRunner.start()
build parameter for JVM running, JVM was selected by JvmManager which obey singleInstance model
entry of JVM is org.apache.hadoop.mapred.Child which will call, which will call Task.run()
in MapTask.run() or ReduceTask.run(), our mapper or reducer code waw call through
**************
MapRunnable runner = ReflectionUtils.newInstance(job
.getMapRunnerClass(), job);
**************
finally, we finish our map/reduce work.
分享到:
评论

相关推荐

    win7下hadoop job提交

    以上就是在Windows 7环境下提交Hadoop Job的全过程,涉及到Java环境配置、Hadoop安装与配置、MapReduce编程、Job提交和监控等多个环节。通过这个过程,你可以更好地理解Hadoop的工作原理和分布式计算的基本概念。

    Job提交流程源码解析.xmind

    Hadoop集群的job提交源码解析思维导图

    Hadoop提交Job执行MR程序代码.zip

    本文将详细解析标题为“Hadoop提交Job执行MR程序代码.zip”的内容,涵盖Hadoop MR程序代码、job提交流程以及MR程序的不同运行模式。 首先,Hadoop MapReduce(简称MR)是基于Google的MapReduce编程模型实现的,主要...

    job提交yarn平台过程.png

    job提交yarn平台过程.png,描述的是hadoop 作业提交过程

    hadoop源码解析-Job提交.pdf

    在提交Job之前,需要将Job的JAR包复制到集群中,这个过程由`copyAndConfigureFiles()`和`uploadFiles()`完成。同时,`writeSplits()`方法负责计算数据的切片,这是通过`FileInputFormat.getSplit()`实现的。这个方法...

    Hadoop源码分析(client端提交job到rm端)

    学习Hadoop源码过程中做的源码分析,共享一下,PPT中有我的邮箱,可以互相探讨。Hadoop源码分析(client端提交job到rm端)

    HADOOP安装过程

    本文档基于一位用户的实践经历,详细介绍了如何在虚拟机环境下安装和配置Hadoop的过程。该文档不仅适用于初学者了解Hadoop的基本安装流程,对于有一定基础的技术人员来说也是一个很好的参考。 #### 二、准备工作 *...

    Linux提交hadoop任务

    JobSubmitter会设置作业参数,如输入输出路径、分区函数、排序规则等,然后调用Hadoop的`Job`类来提交作业。 7. **提交Hadoop任务**:在Linux环境下,开发者通常使用`hadoop jar`命令来提交作业,格式如下: ``` ...

    Hadoop完全分布式详细安装过程

    整个安装过程分为六个主要部分:安装虚拟化工具VMware、在VMware上安装Ubuntu系统、安装JDK与SSH服务作为Hadoop安装前的准备、配置Hadoop、安装Eclipse以及运行一个简单的Hadoop程序——WordCount.java。 #### 二、...

    Hadoop Shuffle过程全解析

    Hadoop Mapreduce过程shuffle过程全解析,Shuffle过程

    Hadoop2.7.7安装过程

    在整个安装过程中,学习者不仅复习了基本的Linux命令,还增加了对Ubuntu系统、软件包管理、环境变量配置以及问题解决策略的理解。遇到困难时,通过查阅网络资源,如百度、B站、知乎和CSDN,学习并应用他人的经验,这...

    Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程

    Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程 按照文档中的操作步骤,一步步操作就可以完全实现hadoop2.2.0版本的完全分布式集群搭建过程

    Hadoop的安装、配置过程和集群运行截图。

    较为完善的Hadoop伪分布式安装、配置过程以及运行截图 全程跟着安装配置的一般不会出现问题,jdk版本尽量选择和Hadoop版本相容的,Hadoop版本可以选择较低版本,2.7版本较为稳定,Linux系统版本没有多大要求,一般将...

    hadoop 分布式部署全过程

    本文将详细介绍 Hadoop 分布式部署全过程,包括安装介质的选择、虚拟机的创建、Linux 操作系统的安装、Hadoop 的安装和配置等步骤,同时也会对常见的问题进行分析和解决。 第一步:安装介质的选择 在安装 Hadoop ...

    Hadoop课程实验和报告——Hadoop安装实验报告

    Hadoop的安装过程可以按照官方文档进行,整个过程包括了在Java JDK和JRE的安装、SSH服务的开启以及Hadoop配置文件的设置。首先,需要下载最新的稳定版本的Hadoop,然后按照官方文档进行安装和配置。 1.1、Java JDK...

    大数据 hadoop mapreduce 词频统计

    4. **运行Job**:配置好MapReduce作业后,提交到Hadoop集群进行执行。集群会自动调度任务,将工作分配给各个节点。 5. **结果收集**:MapReduce完成后,最终的词频统计结果会被写入HDFS,可以进一步进行可视化或...

    基于Java和ssh在Hadoop平台上完成文件操作

    Java API提供了`Job`类,可以用来提交Job、获取Job状态、等待Job完成并获取最终输出。结合SSH,可以在远程节点上执行数据分析脚本,对MapReduce的结果进行进一步处理。 文件名为“ssh_v3-1.1”的压缩包可能包含的是...

Global site tag (gtag.js) - Google Analytics