hadoop 2.x-enable job historyserver

  we know,hadoop will show only abstract  result info about mapreduce(represent in http://xxx:19888/clsuter by default),so it's unconvenient to track the number of mappers,where mappers run,how to track the exception logs etc.note ,this is the default behavior by hadoop(maybe decrease the resouces occupied).

  and yep,there is a deamon to process these logs files,within it u can find what u can see in hadoop-1.x.here are some steps to enable this feature below:

 

  1.add a property in yarn-site.xml

yarn.log-aggregation-enable=true

    note:if u specify tihs ,the containers logs under 'userlogs' will be removed after job completed.

    also,u can specify some relative items:

  

  <property>
    <description>How long to keep aggregation logs before deleting them.  -1 disables. 
    Be careful set this too small and you will spam the name node.</description>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>-1</value>
  </property>
  <property>
    <description>How long to wait between aggregated log retention checks.
    If set to 0 or a negative value then the value is computed as one-tenth
    of the aggregated log retention time. Be careful set this too small and
    you will spam the name node.</description>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
    <value>-1</value>
  </property>

  <property>
    <description>Time in seconds to retain user logs. Only applicable if
    log aggregation is disabled
    </description>
    <name>yarn.nodemanager.log.retain-seconds</name>
    <value>10800</value>
  </property>

  <property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
  </property>
  <property>
    <description>The remote log dir will be created at 
      {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
    </description>
    <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
    <value>logs</value>
  </property>

 

  2. specify the host where the 'JobHistoryServer' to run by mapred-site.xml

 

    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>host:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>host:19888</value>
    </property>

 

 

  3.spawn mr history server

mr-jobhistory-daemon.sh start historyserver

  now u can see a daemon named JobHistoryServer in the host configured by mapreduce.jobhistory.address

 

  after all above complete,u can go to the historyserver by

 

http://host:19888/jobhistory

 

 now some figures are shown here:

 

 if u click the link 'history'(ie.http://host:50030/proxy/application_1418972108758_0001/jobhistory/job/job_1418972108758_0001),then u will be redirected to the job history server 'http://host:19888/jobhistory/job/job_1418972108758_0001/jobhistory/job/job_1418972108758_0001'

 

 

 

 ref:

HistoryServer的原理详解