Win7下用Eclipse向Hadoop2.5.2集群提交MapReduce程序的注意事项

 

主要描述下，在Win7环境下，通过eclipse往集群提交MapReduce程序的过程。

一、环境说明：

开发环境：WIN7

Eclipse版本：eclipse-jee-indigo

Hadoop版本：Hadoop2.5.2

MR运行模式：Yarn

 

二、使用MapReduce的Eclipse插件：

插件名称：hadoop-eclipse-plugin-2.5.2.jar

可以网上下载，本文档附件中也有上传。

将插件复制到${ECLIPSE_HOME}/plugins/目录下。

 

三、运行简单的MR（如：Wordcount）程序，碰到下面问题：

1、运行测试程序时，会出现以下报错：

ExitCodeException exitCode=1: 

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

解决方法：

需要在yarn-site.xml上，配置以下内容：

  <property>

        <name>mapreduce.job.ubertask.enable</name>

        <value>true</value>

  </property>

当在WIN7下用Eclipse提交MR程序时，如果还报这样的错误，需要在客户端代码增加如下配置：

conf.set("mapreduce.job.ubertask.enable","true");

 

2、运行测试程序时，会出现以下报错：

ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

 

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

 

 

Container exited with a non-zero exit code 1

.Failing this attempt.. Failing the application.

解决方法：

conf.set("mapreduce.app-submission.cross-platform", "true");

或：

conf.setBoolean("mapreduce.app-submission.cross-platform", true);

              

3、正常情况下，在Win7下通过Eclipse向集群提交运行程序时，MapReduce的Main方法需要增加以下配置项：

conf.set("fs.default.name", "hdfs://xxx.xxx.xxx.xxx:8020");

conf.set("yarn.resourcemanager.address", "xxx.xxx.xxx.xxx:8032");

conf.set("yarn.resourcemanager.scheduler.address", "xxx.xxx.xxx.xxx:8030");

conf.set("mapreduce.jobhistory.address", "xxx.xxx.xxx.xxx:10020");

conf.set("mapreduce.framework.name", "yarn");

        conf.setBoolean("mapreduce.app-submission.cross-platform", true);

        conf.setBoolean("mapreduce.job.ubertask.enable", true);