`
zy19982004
  • 浏览: 663229 次
  • 性别: Icon_minigender_1
  • 来自: 深圳
博客专栏
F6f66edc-1c1a-3859-b76b-a22e740b7aa7
Hadoop学习
浏览量：252271
社区版块
存档分类
最新评论

Hadoop学习三十二：Win7下无法提交MapReduce Job到集群环境

 
阅读更多

一. 对hadoop eclipse plugin认识不足

      http://zy19982004.iteye.com/blog/2024467曾经说到我最hadoop eclipse plugin作用的认识。但事实上我犯了一个错误，Win7 Eclipse里的MyWordCount程序一直在本地运行，没有提交到集群环境上运行（查看192.168.1.200:50030）没有这个Job。运行方式分为两种，右键Run As

  1. Java Application
  2. Run on Hadoop

     如果说Run As Java Application在本地运行还好说，它直接使用项目下的依赖的Hadoop Jar，使用Hdfs作为input，对MyWordCount main方法的一步步调用，把输出结果写入Hdfs完成。这一切都跟MapReduce集群无关。

     但Run on Hadoop为什么不行呢，难道这个插件的作用仅仅如http://zy19982004.iteye.com/blog/2024467说到的这么简单？

 

二. Hadoop2.x eclispe-plugin

      再次下载源码https://github.com/winghc/hadoop2x-eclipse-plugin。简单看了几个类，如

Wizard for publishing a job to a Hadoop server
public class RunOnHadoopWizard extends Wizard {}


Representation of a Map/Reduce running job on a given location
public class HadoopJob {}

      从注释就可以看出来插件是支持远程提交Job的。那是我使用不当吗？

 

 

三. Hadoop2.x eclispe-plugin工作原理

     Run on Hadoop时

  1. 会在EclipseWorkspace\.metadata\.plugins\org.apache.hadoop.eclipse\下生成一个MapReduce Jar和一个对应文件夹（包含core-site.xml）。我们在Eclipse里配置的Hadoop集群信息会写到core-site.xml里面。
  2. 然后把此Job依据配置信息提交到本地或者集群。

     我去看了下Job对应的core-site.xml，mapreduce.framework.name居然是local，yarn.resourcemanager.address居然是0.0.0.0:8032，于是回到Eclispe配置集群环境的地方，发现果然是这样的，也就是说插件根本没有把集群环境下的配置信息全部copy到Eclipse下。把Eclipse下这两项修改后，还是在本地运行，我就奇怪了，于是我在程序里加上

conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.address", "192.168.1.200:8032");

      终于正常提交到集群环境了。我怀疑插件最后在什么地方还是读取到了local和0.0.0.0:8032，写入了core-site.xml，有时间再去看看插件的源码。

 

四. 继续出问题

     Job虽然是提交到了集群环境，但运行失败了。查看日志如下

2014-04-01 19:50:36,731 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1396351641800_0005_02_000001 : 
%JAVA_HOME% -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA  -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 
2014-03-13 22:50:41,317 INFO  org.apache.hadoop.mapreduce.Job - Job job_1394710790246_0003 failed with state FAILED due to: Application application_1394710790246_0003 failed 2 times due to AM Container for appattempt_1394710790246_0003_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
	at org.apache.hadoop.util.Shell.run(Shell.java:379)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
 

 

      上网找答案，发现是Hadoop本身的问题。

https://issues.apache.org/jira/browse/YARN-1298

https://issues.apache.org/jira/browse/MAPREDUCE-5655。

 

五. 自己编译Hadoop2.2

  1.    下载Hadoop2.2源码http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
  2. 下载https://issues.apache.org/jira/i#browse/MAPREDUCE-5655两个patch
  3. 下载https://issues.apache.org/jira/i#browse/HADOOP-10110这个patch
  4. patch指令。patch -p0 < MRApps.patch。p0的0代表进去层次。不会的参考http://hi.baidu.com/thinkinginlamp/item/0ba1d051319b5ac09e2667f8
  5. 然后按照http://my.oschina.net/yiyuqiuchi/blog/188510去编译。hadoop-2.2.0-src/hadoop-dist/targethadoop-2.2.0.tar.gz就是编译好的。

     给两张patch前后的对照图

     下图左边为patch前Hadoop源码

      下图左边为patch成功后Hadoop源码

 

 

六. 使用自己编译的包

  1.       检查看看patch是否被打包进去了。通过查看MRApps.class字节码得知已经被打包进去。通过查看YARNRunner.class字节码也是正确的，我在YARNRunner.java里面设置了一个PATCH_TEST编译时常量，"zy19982004"的字节码已经被内嵌到class的字节码里。
  2. 替换集群Jar，因为上述三个patch只涉及到两个Jar，另外一个pacth是修改pom文件并且是scope test，可以不管。用hadoop-2.2.0\share\hadoop\mapreduce\hadoop-mapreduce-client-common-2.2.0.jar(MRApps.patch)替换集群下的对应jar，用hadoop-2.2.0\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.2.0.jar(YARNRunner.patch)替换集群下的对应jar。
  3. 修改windows环境下的mapred-site.xml，添加 
    <property>
 <name>mapred.remote.os</name>
 <value>Linux</value>
 <description>Remote MapReduce framework's OS, can be either Linux or Windows</description>
 </property>
      
  4. 重启集群，这个错误已经没有了，但出现了另外的错误。 
    Application application_1396339724108_0014 failed 2 times due to AM Container for appattempt_1396339724108_0014_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:

2014-04-01 19:50:36,731 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1396351641800_0005_02_000001 : 

$JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA  -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
     然后 
    1) Add such config property to your mapred-site.xml (client side only):
<property>
<name>mapreduce.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
</value>
</property>
     
     终于成功了。。。

七. 不使用hadoop eclispe plugin的场景

      自己调试Hadoop源码的时候，Debug As Java Application即可。前面两篇博客解决错误都是通过debug源码来解决的。

 

八. 总结

     Window向Linux Hadoop提交作业的方法

  1. 配置好hadoop eclipse plugin。
  2. Job配置文件里mapreduce.framework.name为yarn。其它配置也需要正确。
  3. Run On Hadoop

     Run As Application其实也可以提交Job，依赖于上一次Run on Hadoop过程中产生的jar，这为我们debug提供了一种思路。

 

4
顶
0
踩
分享到：
| Hadoop学习三十一：Win7下HBase与MapReduc ...
评论
11 楼 zy19982004 2014-12-22  
,
songyuejs 写道
楼主，我按照你的帖子一步一步改到最后，结果还是有错误。我发现作业提交成功了，可是运行出现如下错误：
2014-12-18 17:37:23,630 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2014-12-18 17:37:23,653 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM.
java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=-1, maxMemory=8192
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:198)
        at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:78)
        at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:394)
        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
        at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2callBlockingMethod(ApplicationMasterProtocol.java:99)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
能帮我看看是为什么吗，这种错误我在网上没看到过，真不知道怎么解决，谢谢了

  /**
   * Utility method to validate a resource request, by insuring that the
   * requested memory/vcore is non-negative and not greater than max
   *
   * @throws <code>InvalidResourceRequestException</code> when there is invalid
   *         request
   */
  public static void validateResourceRequest(ResourceRequest resReq,
      Resource maximumResource) throws InvalidResourceRequestException {
    if (resReq.getCapability().getMemory() < 0 ||
        resReq.getCapability().getMemory() > maximumResource.getMemory()) {
      throw new InvalidResourceRequestException("Invalid resource request"
          + ", requested memory < 0"
          + ", or requested memory > max configured"
          + ", requestedMemory=" + resReq.getCapability().getMemory()
          + ", maxMemory=" + maximumResource.getMemory());
    }


你可以自己看看源代码   为什么resReq.getCapability().getMemory()= -1
10 楼 songyuejs 2014-12-18  
楼主，我按照你的帖子一步一步改到最后，结果还是有错误。我发现作业提交成功了，可是运行出现如下错误：
2014-12-18 17:37:23,630 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2014-12-18 17:37:23,653 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM.
java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=-1, maxMemory=8192
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:198)
        at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:78)
        at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:394)
        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
        at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2callBlockingMethod(ApplicationMasterProtocol.java:99)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
能帮我看看是为什么吗，这种错误我在网上没看到过，真不知道怎么解决，谢谢了
9 楼 wsy85 2014-08-13   
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/simba/.staging is not as expected. It is owned by . The directory must be owned by the submitter simba or by simba
 at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:112)
 at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:344)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Unknown Source)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
 at WordCount.main(WordCount.java:111)

在本地运行没问题，但加上你第三步说的那两行代码提交到集群上去执行就报如上异常。simba是我在hadoop集群的那台Linux的用户名，也是我win7的用户名。我重新设置了文件权限，甚至删除/tmp/目录里面的内容都无效。
8 楼 zy19982004 2014-06-27  
gao4263 写道

你的两个问题
1.还是报错“exited with  exitCode: 1 due to: Exception from container-launch” 需要你自己去debug了
2.ClassNotFoundException考虑插件是不是没有把jar包提交到hdfs上，也可以通过debug才看到。
7 楼 gao4263 2014-06-26  
楼主这个问题无解啊：
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Map not found

我运行方式是 run on hadoop
6 楼 gao4263 2014-06-26  
gao4263 写道
楼主 我下载了你的包还是报
14/06/26 15:43:51 INFO mapreduce.Job: Job job_1403768617899_0002 failed with state FAILED due to: Application application_1403768617899_0002 failed 2 times due to AM Container for appattempt_1403768617899_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


1) Add such config property to your mapred-site.xml (client side only): 
<property> 
<name>mapreduce.application.classpath</name> 
<value> 
$HADOOP_CONF_DIR, 
$HADOOP_COMMON_HOME/share/hadoop/common/*, 
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*, 
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*, 
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, 
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, 
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, 
$HADOOP_YARN_HOME/share/hadoop/yarn/*, 
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/* 
</value> 
</property>   这个一定要写才可以啊，
5 楼 gao4263 2014-06-26  
楼主 我下载了你的包还是报
14/06/26 15:43:51 INFO mapreduce.Job: Job job_1403768617899_0002 failed with state FAILED due to: Application application_1403768617899_0002 failed 2 times due to AM Container for appattempt_1403768617899_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4 楼 yuezhichao 2014-06-20  
右键 run on hadoop
3 楼 zy19982004 2014-06-03  
yuezhichao 写道
上面的错误我要怎么修改呀！

WordCount程序怎么运行的？
2 楼 yuezhichao 2014-06-01  
上面的错误我要怎么修改呀！
1 楼 yuezhichao 2014-06-01  
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 12 more

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 12 more

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 12 more

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 12 more

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 12 more

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class test.WordCount$Reduce not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 12 more

Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
at test.WordCount.main(WordCount.java:72)
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

    大数据 hadoop mapreduce 词频统计

    4. **运行Job**：配置好MapReduce作业后，提交到Hadoop集群进行执行。集群会自动调度任务，将工作分配给各个节点。 5. **结果收集**：MapReduce完成后，最终的词频统计结果会被写入HDFS，可以进一步进行可视化或...

    window下eclipse中运行mapreduce程序所需要的Hadoop全部jar包

    在Windows环境下，使用Eclipse开发MapReduce程序时，必须确保所有必要的Hadoop库都被正确引入。这是因为MapReduce是Hadoop生态系统中的核心组件，用于处理分布式计算任务。以下是一些关于如何在Eclipse中配置和使用...

    Hadoop学习总结之四：Map-Reduce过程解析

    ### Hadoop MapReduce任务提交与执行流程解析 #### 一、客户端提交任务 在Hadoop MapReduce框架中，客户端的任务提交是整个MapReduce作业启动的关键步骤。这一过程主要由`JobClient`类中的`runJob(JobConf job)`...

    MapReduce Job集群提交过程源码跟踪及分析

    在Hadoop MapReduce框架中，Job的提交过程是整个分布式计算流程中的关键步骤。这个过程涉及到客户端、JobTracker（在Hadoop 2.x版本中被ResourceManager替代）和TaskTracker（在Hadoop 2.x版本中被NodeManager替代）...

    hadoop文档, hdfs mapreduce,环境搭建，例子介绍等

    ### Hadoop概述与环境搭建详解 #### 一、Hadoop简介 Hadoop是一个开源软件框架，主要用于处理大规模数据集（GB到PB级别）的分布式计算。它最初由Apache基金会开发，现已成为大数据处理领域的基石之一。Hadoop的核心...

    hadoop eclipse mapreduce下开发所有需要用到的JAR包

    总结来说，"hadoop eclipse mapreduce下开发所有需要用到的JAR包"涉及到的是Hadoop MapReduce开发的环境配置，确保正确导入所有必要的依赖库，以便在Eclipse中编写、测试和运行MapReduce程序。这些JAR包是Hadoop生态...

    Hadoop MapReduce

    4. Job Submission and Monitoring：负责提交作业到集群并提供监控作业执行过程的功能。 5. Job Input：处理作业的输入数据，通常是读取HDFS中的数据。 6. Job Output：处理作业的输出数据，通常是将结果写回到HDFS...

    hadoop2.x集群搭建.txt（hdfs和yarn貌似正常，但mapreduce 提交job执行失败，请看我的另一个资源，另一个搭建是成功的）

    根据提供的文件信息，本文将详细解析Hadoop 2.x集群的搭建步骤以及遇到的问题，特别是针对MapReduce提交Job执行失败的情况进行分析。 ### Hadoop 2.x 集群搭建 #### 一、前期准备 在搭建Hadoop 2.x集群之前，我们...

    提高hadoop的mapreduce job效率笔记

    在大数据处理领域，Hadoop MapReduce 是一个至关重要的框架，用于分布式处理海量数据。通过优化 MapReduce Job 的配置和设计，可以显著提升处理效率。以下是一些关键知识点，旨在帮助你提升 Hadoop MapReduce Job 的...

    hadoop eclipse mapreduce 下开发所有需要用到的 JAR 包

    Eclipse与Hadoop的集成使得开发者可以在本地环境中编写、测试和调试MapReduce作业，然后再将其部署到Hadoop集群上运行。 要开始在Eclipse中开发MapReduce项目，首先你需要安装以下关键的JAR包，这些包提供了Hadoop ...

    Hadoop MapReduce实现tfidf源码

    此外，还需要配置Job参数，如输入路径、输出路径、Mapper和Reducer类等，并提交Job到Hadoop集群执行。 压缩包中的"tfidf"文件可能是包含源代码、测试数据或者运行结果的文件。源代码可能包含了Mapper和Reducer的...

    传播智客-Hadoop实战视频教程下载：里面还包括环境搭建所需的文件、教程

    【Hadoop实战视频教程】是针对大数据处理领域的一款专业学习资源，主要涵盖了Hadoop的各个方面，包括环境搭建、核心组件的使用以及实际案例的应用。这个教程不仅提供了视频讲解，还包含有配套的环境搭建文件，使得...

    安装Hadoop（单机版和集群）步骤

    10. **配置MapReduce参数**：编辑`mapred-site.xml`，设置JobTracker地址和端口，集群模式下需改为其他节点IP： ``` &lt;name&gt;mapred.job.tracker &lt;value&gt;localhost:9001 ``` 11. **格式化NameNode**：启动...

    hadoop mapreduce helloworld 能调试

    在大数据处理领域，Hadoop MapReduce 是一个至关重要的框架，它允许开发者编写分布式应用程序来处理海量数据。"Hadoop MapReduce HelloWorld 能调试" 的主题意味着我们将深入理解如何设置、运行以及调试 MapReduce ...

    大数据实验5实验报告：MapReduce 初级编程实践

    通过Hadoop的`Job`类和相关输入输出格式类，可以设置这些参数并提交作业到Hadoop集群执行。 总结起来，这个实验不仅介绍了MapReduce的基本概念，还展示了如何用Java实现MapReduce任务，以解决实际问题。通过这种...

    最新Hadoop学习笔记

    本笔记将从环境搭建、HDFS配置、YARN配置、分布式配置到MapReduce编程，全方位介绍Hadoop的学习过程，并结合JavaWeb项目实践，帮助读者深入理解Hadoop的工作原理和应用。 ### 一、环境搭建 在开始Hadoop的学习之前...

    第一个Mapreduce程序.pdf

    文中没有详细说明如何运行作业，但在Hadoop MapReduce中，一般通过Hadoop命令行工具来提交作业，命令通常类似于“hadoop jar wc3.jar”。 整个过程大致如下： 1. 搭建Hadoop环境（以CDH5为基础）。 2. 编写或获取...

    如何在Windows下的eclipse调试Hadoop2.2.0分布式集群

    在Windows环境下，使用Eclipse进行Hadoop 2.2.0分布式集群的调试是一项重要的技能，这可以帮助开发者更好地理解和优化Hadoop程序。以下是一些关键的知识点，将指导你完成这个过程。 首先，Hadoop是一个开源的分布式...

    Java访问Hadoop集群源码

    在Java编程环境中，访问Hadoop集群是一项常见的任务，特别是在大数据处理和分析的场景下。Hadoop是一个开源框架，主要用于存储和处理大规模数据集。本文将深入探讨如何利用Java API来与Hadoop集群进行交互，包括读取...

Magicbox
Global site tag (gtag.js) - Google Analytics