`
weitao1026
  • 浏览: 1053211 次
  • 性别: Icon_minigender_1
  • 来自: 上海
社区版块
存档分类
最新评论

在Win上提交hadoop集群的作业

 
阅读更多

一直以来，都以为，想在Win上提交hadoop集群的作业，必须得在eclipse上安装hadoop-eclipse-plugin插件才可以提交，但最近与同事交流，发现其实，不一定必须安装hadoop的eclipse插件，才能提交。今天试了一把，发现果然可以不用安装插件也可以正确提交作业到集群上，故在此总结一下。


既然，无须安装hadoop的eclipse插件，就能提交hadoop作业，那为毛，还出现了这个插件呢？   其实安装插件除了能直接提交作业外，还有一个比较方便的功能，就是能直接在eclipse上对HDFS上的文件，进行删除，上传，新建目录等，这一点是不安装插件做不到的，当然，如果你不需要这些操作，那么就无所谓了，仅仅提交个作业而已。

 

下面说下，如何在eclipse上使用无插件提交hadoop作业，（在hadoop集群的8088界面上可以看到提交的作业信息是否成功）。

序号 操作 说明 1 eclipse IDE 散仙在这里是4.2版本的eclipse 2 hadoop2.2的64位完整包 散仙在这里放在D盘根目录下 3 修改源码org/apache/hadoop/mapred/YARNRunner.java，改变linux与windows的路径不一致bug 散仙已经修改好，文末散仙会上传这个修改好的类 4 把linux集群上的配置文件，core-site.xml，hdfs-site.xml,mapred.site.xml和yarn-site.xml文件，放在src根目录下，另外在D盘hadoop的/etc/hadoop目录下，覆盖一下 注意一致 5 编写wordcount的MR例子，开始测试 入门测试 6 高富帅工程师一名 主角 7 配置hadoop的win上的环境变量HADOOP_HOME 只配置这一个即可

 



上面的操作都完成后，就可以进行测试了,散仙在这里的WordCount源码如下：

Java代码 复制代码 收藏代码
  1. package com.mywordcount;  
  2.   
  3.    
  4.   
  5.    
  6. import java.io.File;  
  7. import java.io.FileInputStream;  
  8. import java.io.FileNotFoundException;  
  9. import java.io.FilenameFilter;  
  10. import java.io.IOException;  
  11.   
  12. import org.apache.hadoop.conf.Configuration;  
  13. import org.apache.hadoop.conf.Configured;  
  14. import org.apache.hadoop.fs.FileSystem;  
  15. import org.apache.hadoop.fs.Path;  
  16. import org.apache.hadoop.io.IntWritable;  
  17. import org.apache.hadoop.io.LongWritable;  
  18. import org.apache.hadoop.io.Text;  
  19. import org.apache.hadoop.mapred.JobConf;  
  20. import org.apache.hadoop.mapreduce.Job;  
  21. import org.apache.hadoop.mapreduce.Mapper;  
  22. import org.apache.hadoop.mapreduce.Reducer;  
  23. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
  24. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;  
  25. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
  26. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;  
  27. import org.apache.hadoop.util.Tool;  
  28.   
  29. /*** 
  30.  *  
  31.  * Hadoop2.2.0  无插件提交集群作业 
  32.  *  
  33.  * @author qindongliang 
  34.  *  
  35.  *         hadoop技术交流群： 376932160 
  36.  *  
  37.  *  
  38.  * */  
  39. public class MyWordCount2 {  
  40.   
  41.     /** 
  42.      * Mapper 
  43.      *  
  44.      * **/  
  45.     private static class WMapper extends  
  46.             Mapper<LongWritable, Text, Text, IntWritable> {  
  47.   
  48.         private IntWritable count = new IntWritable(1);  
  49.         private Text text = new Text();  
  50.   
  51.         @Override  
  52.         protected void map(LongWritable key, Text value, Context context)  
  53.                 throws IOException, InterruptedException {  
  54.             String values[] = value.toString().split("#");  
  55.             // System.out.println(values[0]+"========"+values[1]);  
  56.             count.set(Integer.parseInt(values[1]));  
  57.             text.set(values[0]);  
  58.             context.write(text, count);  
  59.   
  60.         }  
  61.   
  62.     }  
  63.   
  64.     /** 
  65.      * Reducer 
  66.      *  
  67.      * **/  
  68.     private static class WReducer extends  
  69.             Reducer<Text, IntWritable, Text, Text> {  
  70.   
  71.         private Text t = new Text();  
  72.   
  73.         @Override  
  74.         protected void reduce(Text key, Iterable<IntWritable> value,  
  75.                 Context context) throws IOException, InterruptedException {  
  76.             int count = 0;  
  77.             for (IntWritable i : value) {  
  78.                 count += i.get();  
  79.             }  
  80.             t.set(count + "");  
  81.             context.write(key, t);  
  82.   
  83.         }  
  84.   
  85.     }  
  86.   
  87.     public static void printEnv(Job job) {  
  88.         Configuration conf = job.getConfiguration();  
  89.         System.out.println("###########################################");  
  90.         System.out.println("fs.defaultFS:" + conf.get("fs.defaultFS"));  
  91.         System.out.println("mapred.job.tracker:"  
  92.                 + conf.get("mapred.job.tracker"));  
  93.         System.out.println("mapreduce.framework.name" + ":"  
  94.                 + conf.get("mapreduce.framework.name"));  
  95.         System.out.println("yarn.nodemanager.aux-services" + ":"  
  96.                 + conf.get("yarn.nodemanager.aux-services"));  
  97.         System.out.println("yarn.resourcemanager.address" + ":"  
  98.                 + conf.get("yarn.resourcemanager.address"));  
  99.         System.out.println("yarn.resourcemanager.scheduler.address" + ":"  
  100.                 + conf.get("yarn.resourcemanager.scheduler.address"));  
  101.         System.out.println("yarn.resourcemanager.resource-tracker.address"  
  102.                 + ":"  
  103.                 + conf.get("yarn.resourcemanager.resource-tracker.address"));  
  104.         System.out.println("yarn.application.classpath" + ":"  
  105.                 + conf.get("yarn.application.classpath"));  
  106.         System.out.println("zkhost:" + conf.get("zkhost"));  
  107.         System.out.println("namespace:" + conf.get("namespace"));  
  108.         System.out.println("project:" + conf.get("project"));  
  109.         System.out.println("collection:" + conf.get("collection"));  
  110.         System.out.println("shard:" + conf.get("shard"));  
  111.         System.out.println("###########################################");  
  112.     }  
  113.      /** 
  114.       * 载入hadoop的配置文件 
  115.       * 兼容hadoop1.x和hadoop2.x 
  116.       *  
  117.       * */  
  118.     public static void getConf(final Configuration conf) throws FileNotFoundException{  
  119.         String HADOOP_CONF_DIR = System.getenv().get("HADOOP_CONF_DIR");  
  120.         String HADOOP_HOME = System.getenv().get("HADOOP_HOME");  
  121.         System.out.println("HADOOP_HOME:" + HADOOP_HOME);  
  122.         System.out.println("HADOOP_CONF_DIR:" + HADOOP_CONF_DIR);//此处兼容hadoop1.x  
  123.           
  124.         //此处兼容hadoop2.x  
  125.         if (HADOOP_CONF_DIR == null || HADOOP_CONF_DIR.isEmpty()) {  
  126.             HADOOP_CONF_DIR = HADOOP_HOME + "/etc/hadoop";  
  127.         }  
  128.   
  129.         //得到hadoop的conf目录的路径加载文件  
  130.         File file = new File(HADOOP_CONF_DIR);  
  131.         FilenameFilter filter = new FilenameFilter() {  
  132.   
  133.             @Override  
  134.             public boolean accept(File dir, String name) {  
  135.                 return name.endsWith("xml");  
  136.             }  
  137.         };  
  138.           
  139.           
  140.         //获取hadoop的仅仅xml结尾的文件列表  
  141.         String[] list = file.list(filter);  
  142.         for (String fn : list) {  
  143.             System.out.println("Loading Configuration: " + HADOOP_CONF_DIR  
  144.                     + "/" + fn);  
  145.             //循环加载xml文件  
  146.             conf.addResource(new FileInputStream(HADOOP_CONF_DIR + "/" + fn));  
  147.         }  
  148.   
  149.            
  150.           
  151.         //yarn的classpath路径，如果为空则加载拼接yarn的路径  
  152.         if (conf.get("yarn.application.classpath", "").isEmpty()) {  
  153.             StringBuilder sb = new StringBuilder();  
  154.             sb.append(System.getenv("CLASSPATH")).append(":");  
  155.             sb.append(HADOOP_HOME).append("/share/hadoop/common/lib/*")  
  156.                     .append(":");  
  157.             sb.append(HADOOP_HOME).append("/share/hadoop/common/*").append(":");  
  158.             sb.append(HADOOP_HOME).append("/share/hadoop/hdfs/*").append(":");  
  159.             sb.append(HADOOP_HOME).append("/share/hadoop/mapreduce/*")  
  160.                     .append(":");  
  161.             sb.append(HADOOP_HOME).append("/share/hadoop/yarn/*").append(":");  
  162.             sb.append(HADOOP_HOME).append("/lib/*").append(":");  
  163.             conf.set("yarn.application.classpath", sb.toString());  
  164.         }  
  165.           
  166.           
  167.           
  168.           
  169.           
  170.           
  171.     }  
  172.       
  173.    
  174.   
  175.     public static void main(String[] args) throws Exception { {  
  176.                
  177.             Configuration conf = new Configuration();  
  178.             conf.set("mapreduce.job.jar", "myjob.jar");//此处代码，一定放在Job任务前面，否则会报类找不到的异常  
  179.             Job job = Job.getInstance(conf, "345");    
  180.             getConf(conf);  
  181.             job.setJarByClass(MyWordCount2.class);  
  182.   
  183.             job.setMapperClass(WMapper.class);  
  184.             job.setReducerClass(WReducer.class);  
  185.             job.setInputFormatClass(TextInputFormat.class);  
  186.             job.setOutputFormatClass(TextOutputFormat.class);  
  187.   
  188.             job.setMapOutputKeyClass(Text.class);  
  189.             job.setMapOutputValueClass(IntWritable.class);  
  190.             job.setOutputKeyClass(Text.class);  
  191.             job.setOutputValueClass(Text.class);  
  192.   
  193.             String path = "/qin/output";  
  194.             FileSystem fs = FileSystem.get(conf);  
  195.             Path p = new Path(path);  
  196.             if (fs.exists(p)) {  
  197.                 fs.delete(p, true);  
  198.                 System.out.println("输出路径存在，已删除！");  
  199.             }  
  200.             FileInputFormat.setInputPaths(job, "/qin/input");  
  201.             FileOutputFormat.setOutputPath(job, p);  
  202.             printEnv(job);  
  203.             System.exit(job.waitForCompletion(true) ? 0 : 1);   
  204.            
  205.     }  
  206.   
  207.     }  
  208. }  
package com.mywordcount;

 

 
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FilenameFilter;
import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;

/***
 * 
 * Hadoop2.2.0  无插件提交集群作业
 * 
 * @author qindongliang
 * 
 *         hadoop技术交流群： 376932160
 * 
 * 
 * */
public class MyWordCount2 {

	/**
	 * Mapper
	 * 
	 * **/
	private static class WMapper extends
			Mapper<LongWritable, Text, Text, IntWritable> {

		private IntWritable count = new IntWritable(1);
		private Text text = new Text();

		@Override
		protected void map(LongWritable key, Text value, Context context)
				throws IOException, InterruptedException {
			String values[] = value.toString().split("#");
			// System.out.println(values[0]+"========"+values[1]);
			count.set(Integer.parseInt(values[1]));
			text.set(values[0]);
			context.write(text, count);

		}

	}

	/**
	 * Reducer
	 * 
	 * **/
	private static class WReducer extends
			Reducer<Text, IntWritable, Text, Text> {

		private Text t = new Text();

		@Override
		protected void reduce(Text key, Iterable<IntWritable> value,
				Context context) throws IOException, InterruptedException {
			int count = 0;
			for (IntWritable i : value) {
				count += i.get();
			}
			t.set(count + "");
			context.write(key, t);

		}

	}

	public static void printEnv(Job job) {
		Configuration conf = job.getConfiguration();
		System.out.println("###########################################");
		System.out.println("fs.defaultFS:" + conf.get("fs.defaultFS"));
		System.out.println("mapred.job.tracker:"
				+ conf.get("mapred.job.tracker"));
		System.out.println("mapreduce.framework.name" + ":"
				+ conf.get("mapreduce.framework.name"));
		System.out.println("yarn.nodemanager.aux-services" + ":"
				+ conf.get("yarn.nodemanager.aux-services"));
		System.out.println("yarn.resourcemanager.address" + ":"
				+ conf.get("yarn.resourcemanager.address"));
		System.out.println("yarn.resourcemanager.scheduler.address" + ":"
				+ conf.get("yarn.resourcemanager.scheduler.address"));
		System.out.println("yarn.resourcemanager.resource-tracker.address"
				+ ":"
				+ conf.get("yarn.resourcemanager.resource-tracker.address"));
		System.out.println("yarn.application.classpath" + ":"
				+ conf.get("yarn.application.classpath"));
		System.out.println("zkhost:" + conf.get("zkhost"));
		System.out.println("namespace:" + conf.get("namespace"));
		System.out.println("project:" + conf.get("project"));
		System.out.println("collection:" + conf.get("collection"));
		System.out.println("shard:" + conf.get("shard"));
		System.out.println("###########################################");
	}
	 /**
	  * 载入hadoop的配置文件
	  * 兼容hadoop1.x和hadoop2.x
	  * 
	  * */
	public static void getConf(final Configuration conf) throws FileNotFoundException{
		String HADOOP_CONF_DIR = System.getenv().get("HADOOP_CONF_DIR");
		String HADOOP_HOME = System.getenv().get("HADOOP_HOME");
		System.out.println("HADOOP_HOME:" + HADOOP_HOME);
		System.out.println("HADOOP_CONF_DIR:" + HADOOP_CONF_DIR);//此处兼容hadoop1.x
		
		//此处兼容hadoop2.x
		if (HADOOP_CONF_DIR == null || HADOOP_CONF_DIR.isEmpty()) {
			HADOOP_CONF_DIR = HADOOP_HOME + "/etc/hadoop";
		}

		//得到hadoop的conf目录的路径加载文件
		File file = new File(HADOOP_CONF_DIR);
		FilenameFilter filter = new FilenameFilter() {

			@Override
			public boolean accept(File dir, String name) {
				return name.endsWith("xml");
			}
		};
		
		
		//获取hadoop的仅仅xml结尾的文件列表
		String[] list = file.list(filter);
		for (String fn : list) {
			System.out.println("Loading Configuration: " + HADOOP_CONF_DIR
					+ "/" + fn);
			//循环加载xml文件
			conf.addResource(new FileInputStream(HADOOP_CONF_DIR + "/" + fn));
		}

		 
		
		//yarn的classpath路径，如果为空则加载拼接yarn的路径
		if (conf.get("yarn.application.classpath", "").isEmpty()) {
			StringBuilder sb = new StringBuilder();
			sb.append(System.getenv("CLASSPATH")).append(":");
			sb.append(HADOOP_HOME).append("/share/hadoop/common/lib/*")
					.append(":");
			sb.append(HADOOP_HOME).append("/share/hadoop/common/*").append(":");
			sb.append(HADOOP_HOME).append("/share/hadoop/hdfs/*").append(":");
			sb.append(HADOOP_HOME).append("/share/hadoop/mapreduce/*")
					.append(":");
			sb.append(HADOOP_HOME).append("/share/hadoop/yarn/*").append(":");
			sb.append(HADOOP_HOME).append("/lib/*").append(":");
			conf.set("yarn.application.classpath", sb.toString());
		}
		
		
		
		
		
		
	}
	
 

	public static void main(String[] args) throws Exception { {
			 
			Configuration conf = new Configuration();
			conf.set("mapreduce.job.jar", "myjob.jar");//此处代码，一定放在Job任务前面，否则会报类找不到的异常
			Job job = Job.getInstance(conf, "345");	 
			getConf(conf);
			job.setJarByClass(MyWordCount2.class);

			job.setMapperClass(WMapper.class);
			job.setReducerClass(WReducer.class);
			job.setInputFormatClass(TextInputFormat.class);
			job.setOutputFormatClass(TextOutputFormat.class);

			job.setMapOutputKeyClass(Text.class);
			job.setMapOutputValueClass(IntWritable.class);
			job.setOutputKeyClass(Text.class);
			job.setOutputValueClass(Text.class);

			String path = "/qin/output";
			FileSystem fs = FileSystem.get(conf);
			Path p = new Path(path);
			if (fs.exists(p)) {
				fs.delete(p, true);
				System.out.println("输出路径存在，已删除！");
			}
			FileInputFormat.setInputPaths(job, "/qin/input");
			FileOutputFormat.setOutputPath(job, p);
			printEnv(job);
			System.exit(job.waitForCompletion(true) ? 0 : 1); 
		 
	}

	}
}



项目结构目录，截图如下：



运行信息如下：

Java代码 复制代码 收藏代码
  1. HADOOP_HOME:D:\hadoop-2.2.0  
  2. HADOOP_CONF_DIR:null  
  3. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/capacity-scheduler.xml  
  4. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/core-site.xml  
  5. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/hadoop-policy.xml  
  6. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/hdfs-site.xml  
  7. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/httpfs-site.xml  
  8. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/mapred-site.xml  
  9. Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/yarn-site.xml  
  10. 2014-06-25 20:40:08,419 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2172)) - java.io.FileInputStream@3ba08dab:an attempt to override final parameter: mapreduce.jobtracker.address;  Ignoring.  
  11. 输出路径存在，已删除！  
  12. ###########################################  
  13. fs.defaultFS:hdfs://h1:9000  
  14. 2014-06-25 20:40:08,897 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address  
  15. mapred.job.tracker:h1:8021  
  16. mapreduce.framework.name:yarn  
  17. yarn.nodemanager.aux-services:mapreduce_shuffle  
  18. yarn.resourcemanager.address:h1:8032  
  19. yarn.resourcemanager.scheduler.address:h1:8030  
  20. yarn.resourcemanager.resource-tracker.address:h1:8031  
  21. yarn.application.classpath:$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*  
  22. zkhost:null  
  23. namespace:null  
  24. project:null  
  25. collection:null  
  26. shard:null  
  27. ###########################################  
  28. 2014-06-25 20:40:08,972 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(56)) - Connecting to ResourceManager at h1/192.168.46.32:8032  
  29. 2014-06-25 20:40:09,153 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(149)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.  
  30. 2014-06-25 20:40:09,331 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(287)) - Total input paths to process : 1  
  31. 2014-06-25 20:40:09,402 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(394)) - number of splits:1  
  32. 2014-06-25 20:40:09,412 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - user.name is deprecated. Instead, use mapreduce.job.user.name  
  33. 2014-06-25 20:40:09,412 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.jar is deprecated. Instead, use mapreduce.job.jar  
  34. 2014-06-25 20:40:09,413 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class  
  35. 2014-06-25 20:40:09,413 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class  
  36. 2014-06-25 20:40:09,413 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class  
  37. 2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name  
  38. 2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class  
  39. 2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class  
  40. 2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir  
  41. 2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir  
  42. 2014-06-25 20:40:09,415 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class  
  43. 2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps  
  44. 2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class  
  45. 2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class  
  46. 2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir  
  47. 2014-06-25 20:40:09,502 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(477)) - Submitting tokens for job: job_1403723552088_0016  
  48. 2014-06-25 20:40:09,651 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(174)) - Submitted application application_1403723552088_0016 to ResourceManager at h1/192.168.46.32:8032  
  49. 2014-06-25 20:40:09,683 INFO  [main] mapreduce.Job (Job.java:submit(1272)) - The url to track the job: http://h1:8088/proxy/application_1403723552088_0016/  
  50. 2014-06-25 20:40:09,683 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1317)) - Running job: job_1403723552088_0016  
  51. 2014-06-25 20:40:17,070 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1338)) - Job job_1403723552088_0016 running in uber mode : false  
  52. 2014-06-25 20:40:17,072 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 0% reduce 0%  
  53. 2014-06-25 20:40:23,232 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 100% reduce 0%  
  54. 2014-06-25 20:40:30,273 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 100% reduce 100%  
  55. 2014-06-25 20:40:30,289 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1356)) - Job job_1403723552088_0016 completed successfully  
  56. 2014-06-25 20:40:30,403 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1363)) - Counters: 43  
  57.     File System Counters  
  58.         FILE: Number of bytes read=58  
  59.         FILE: Number of bytes written=160123  
  60.         FILE: Number of read operations=0  
  61.         FILE: Number of large read operations=0  
  62.         FILE: Number of write operations=0  
  63.         HDFS: Number of bytes read=136  
  64.         HDFS: Number of bytes written=27  
  65.         HDFS: Number of read operations=6  
  66.         HDFS: Number of large read operations=0  
  67.         HDFS: Number of write operations=2  
  68.     Job Counters   
  69.         Launched map tasks=1  
  70.         Launched reduce tasks=1  
  71.         Data-local map tasks=1  
  72.         Total time spent by all maps in occupied slots (ms)=4398  
  73.         Total time spent by all reduces in occupied slots (ms)=4263  
  74.     Map-Reduce Framework  
  75.         Map input records=4  
  76.         Map output records=4  
  77.         Map output bytes=44  
  78.         Map output materialized bytes=58  
  79.         Input split bytes=98  
  80.         Combine input records=0  
  81.         Combine output records=0  
  82.         Reduce input groups=3  
  83.         Reduce shuffle bytes=58  
  84.         Reduce input records=4  
  85.         Reduce output records=3  
  86.         Spilled Records=8  
  87.         Shuffled Maps =1  
  88.         Failed Shuffles=0  
  89.         Merged Map outputs=1  
  90.         GC time elapsed (ms)=94  
  91.         CPU time spent (ms)=980  
  92.         Physical memory (bytes) snapshot=310431744  
  93.         Virtual memory (bytes) snapshot=1681850368  
  94.         Total committed heap usage (bytes)=136450048  
  95.     Shuffle Errors  
  96.         BAD_ID=0  
  97.         CONNECTION=0  
  98.         IO_ERROR=0  
  99.         WRONG_LENGTH=0  
  100.         WRONG_MAP=0  
  101.         WRONG_REDUCE=0  
  102.     File Input Format Counters   
  103.         Bytes Read=38  
  104.     File Output Format Counters   
  105.         Bytes Written=27  
HADOOP_HOME:D:\hadoop-2.2.0
HADOOP_CONF_DIR:null
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/capacity-scheduler.xml
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/core-site.xml
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/hadoop-policy.xml
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/hdfs-site.xml
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/httpfs-site.xml
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/mapred-site.xml
Loading Configuration: D:\hadoop-2.2.0/etc/hadoop/yarn-site.xml
2014-06-25 20:40:08,419 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2172)) - java.io.FileInputStream@3ba08dab:an attempt to override final parameter: mapreduce.jobtracker.address;  Ignoring.
输出路径存在，已删除！
###########################################
fs.defaultFS:hdfs://h1:9000
2014-06-25 20:40:08,897 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
mapred.job.tracker:h1:8021
mapreduce.framework.name:yarn
yarn.nodemanager.aux-services:mapreduce_shuffle
yarn.resourcemanager.address:h1:8032
yarn.resourcemanager.scheduler.address:h1:8030
yarn.resourcemanager.resource-tracker.address:h1:8031
yarn.application.classpath:$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
zkhost:null
namespace:null
project:null
collection:null
shard:null
###########################################
2014-06-25 20:40:08,972 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(56)) - Connecting to ResourceManager at h1/192.168.46.32:8032
2014-06-25 20:40:09,153 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(149)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2014-06-25 20:40:09,331 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(287)) - Total input paths to process : 1
2014-06-25 20:40:09,402 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(394)) - number of splits:1
2014-06-25 20:40:09,412 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - user.name is deprecated. Instead, use mapreduce.job.user.name
2014-06-25 20:40:09,412 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-06-25 20:40:09,413 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
2014-06-25 20:40:09,413 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
2014-06-25 20:40:09,413 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name
2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-06-25 20:40:09,414 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2014-06-25 20:40:09,415 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
2014-06-25 20:40:09,416 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
2014-06-25 20:40:09,502 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(477)) - Submitting tokens for job: job_1403723552088_0016
2014-06-25 20:40:09,651 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(174)) - Submitted application application_1403723552088_0016 to ResourceManager at h1/192.168.46.32:8032
2014-06-25 20:40:09,683 INFO  [main] mapreduce.Job (Job.java:submit(1272)) - The url to track the job: http://h1:8088/proxy/application_1403723552088_0016/
2014-06-25 20:40:09,683 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1317)) - Running job: job_1403723552088_0016
2014-06-25 20:40:17,070 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1338)) - Job job_1403723552088_0016 running in uber mode : false
2014-06-25 20:40:17,072 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 0% reduce 0%
2014-06-25 20:40:23,232 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 100% reduce 0%
2014-06-25 20:40:30,273 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 100% reduce 100%
2014-06-25 20:40:30,289 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1356)) - Job job_1403723552088_0016 completed successfully
2014-06-25 20:40:30,403 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1363)) - Counters: 43
	File System Counters
		FILE: Number of bytes read=58
		FILE: Number of bytes written=160123
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=136
		HDFS: Number of bytes written=27
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=4398
		Total time spent by all reduces in occupied slots (ms)=4263
	Map-Reduce Framework
		Map input records=4
		Map output records=4
		Map output bytes=44
		Map output materialized bytes=58
		Input split bytes=98
		Combine input records=0
		Combine output records=0
		Reduce input groups=3
		Reduce shuffle bytes=58
		Reduce input records=4
		Reduce output records=3
		Spilled Records=8
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=94
		CPU time spent (ms)=980
		Physical memory (bytes) snapshot=310431744
		Virtual memory (bytes) snapshot=1681850368
		Total committed heap usage (bytes)=136450048
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=38
	File Output Format Counters 
		Bytes Written=27



至此，我们已经可以成功的在无插件的环境里提交hadoop任务了，如果提交过程中，出现权限异常，可以在eclipse的run环境里配置，linux上安装hadoop的用户名即可，截图如下：



注意，一定是安装hadoop的用户，写成其他的用户，可能会导致没有权限访问HDFS上的数据，从而使提交的作业运行失败。

分享到：
| 把eclipse中的hadoop项目，换了一个工作空 ...
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

    hadoop2.6(x64)Win7上远程调试hadoop 集群

    - 当作业运行时，IDE将自动连接到运行中的Java进程，从而可以在Win7上进行远程调试。 8. **监控和故障排查**： - 使用Hadoop自带的Web UI监控集群状态，包括NameNode、ResourceManager和NodeManager的Web界面。 ...

    win7下编译过的hadoop2.7.2

    Hadoop是Apache软件基金会开发的一个开源分布式计算框架，版本2.7.2是其一个重要的稳定版本。在Windows 7操作系统上...在使用过程中，确保遵循Hadoop的配置指南，以便在Windows环境中正确地设置和管理你的Hadoop集群。

    win7下编译过的hadoop jar包--hadoop-2.7.2.zip

    10. **运行MapReduce程序**：将Java MapReduce程序打包成jar文件，使用`hadoop jar`命令提交作业到Hadoop集群执行。 在Windows 7环境下运行Hadoop可能遇到的问题包括权限问题、路径分隔符差异、文件系统不兼容等。...

    win10下hadoop2.6.4 bin(含winutils.exe和hadoop.dll).zip

    MapReduce作业可以通过Hadoop的命令行工具提交到运行中的HDFS和MapReduce集群。 7. **Hadoop生态系统**: Hadoop不仅仅是HDFS和MapReduce，还包括Pig、Hive、HBase等工具。在Windows上，这些组件可能需要额外的配置...

    win10_64_hadoop

    对于初次使用者，可能需要查阅官方文档或在线教程来学习如何在Windows 10上设置和管理Hadoop集群。 总之，"win10_64_hadoop"提供了一种便捷的方式，在Windows 10系统中快速部署和运行Hadoop，免去了手动编译和解决...

    win7下eclipse配置hadoop的插件

    6. **编写和运行MapReduce程序**：现在，你已经可以在Eclipse中编写Java代码实现MapReduce任务，并直接在Eclipse内提交到Hadoop集群运行，无需离开IDE。 在实际操作中，可能还会遇到其他问题，比如JVM版本不兼容、...

    Hadoop-2.7.4-win-bin

    这些命令行工具允许用户与Hadoop集群进行交互，执行如启动、停止服务、上传数据、运行MapReduce作业等操作。在Windows环境下，这些工具通常需要通过命令提示符或PowerShell来运行。 安装和配置Hadoop在Windows上...

    hadoop-2.7.2_win.zip

    这个版本包含了运行Hadoop所需的所有组件和配置文件，以便在Windows系统上搭建Hadoop集群或单机模式。 1. **主要组件**： - **bin**：包含Hadoop的可执行脚本，如`hadoop`, `hdfs`, `yarn`等，用户可以通过这些...

    hadoop2.7.0版本的hadoop.dll，winutils.exe，X64下编译

    在Windows下开发MapReduce程序，你需要了解Java编程基础，因为MapReduce作业通常是用Java编写并提交到Hadoop集群的。理解Map和Reduce的概念，以及如何编写Mapper和Reducer类，是MapReduce编程的关键。你还需要学习...

    windows平台使用hadoop hdfs文件进行中文分词的示例代码

    接着，你需要安装Hadoop HDFS客户端，这允许你在Windows上与Hadoop集群交互。可以使用Apache Hadoop的Winutils工具包来实现这一点，它提供了与HDFS通信所需的命令行工具。 在Eclipse中，创建一个新的MapReduce项目...

    hadoop入门教程

    4. **提交作业到Hadoop集群**: 编译并打包项目后，通过Eclipse提交作业到Hadoop集群进行执行。 通过以上步骤，不仅能够在Ubuntu环境下成功搭建Hadoop集群，还可以利用Eclipse开发工具实现Hadoop应用程序的开发与...

    在win7编译过的haddop

    3. **监控与管理**：使用Hadoop自带的Web UI或命令行工具监控Hadoop集群的状态，包括DFS和YARN的健康状况。 **注意事项：** - Windows环境下运行Hadoop可能存在性能瓶颈，因为它不是Hadoop的最佳运行平台。生产环境...

    Azkaban3.13变异后安装版

    5. **azkaban-hadoop-security-plugin-3.13.0-6-gd32d0d9.zip**：这个插件提供了对Hadoop的安全支持，如Kerberos认证，这对于在安全的Hadoop集群上运行Azkaban至关重要。 6. **azkaban-sql-3.13.0-6-gd32d0d9.zip**...

    azkaban.rar

    Azkaban是一款开源的工作流管理工具，主要用于大数据处理工作流的调度和执行。...通过解压和部署提供的压缩文件，用户可以在本地或集群环境中搭建起一个高效、可靠的Azkaban实例，以满足其大数据处理的需求。

Magicbox
Global site tag (gtag.js) - Google Analytics