已经讲了Hadoop的单机伪分布的部署，本篇，散仙就说下，如何eclipse中调试hadoop2.2.0,如果你使用的还是hadoop1.x的版本，那么，也没事，散仙在以前的博客里，也写过eclipse调试1.x的hadoop程序，两者最大的不同之处在于使用的eclipse插件不同，hadoop2.x与hadoop1.x的API，不太一致，所以插件也不一样，我们只需要使用分别对应的插件即可.
下面开始进入正题:
| 序号 | 名称 | 描述 | 1 | eclipse | Juno Service Release 4.2的本 | 2 | 操作系统 | Windows7 | 3 | hadoop的eclipse插件 | hadoop-eclipse-plugin-2.2.0.jar | 4 | hadoop的集群环境 | 虚拟机Linux的Centos6.5单机伪分布式 | 5 | 调试程序 | Hellow World |
遇到的几个问题如下：
Java代码 
- java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
解决办法:
在org.apache.hadoop.util.Shell类的checkHadoopHome()方法的返回值里写固定的
本机hadoop的路径，散仙在这里更改如下：
Java代码
- private static String checkHadoopHome() {
- // first check the Dflag hadoop.home.dir with JVM scope
- //System.setProperty("hadoop.home.dir", "...");
- String home = System.getProperty("hadoop.home.dir");
- // fall back to the system/user-global env variable
- if (home == null) {
- home = System.getenv("HADOOP_HOME");
- }
- try {
- // couldn't find either setting for hadoop's home directory
- if (home == null) {
- throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
- }
- if (home.startsWith("\"") && home.endsWith("\"")) {
- home = home.substring(1, home.length()-1);
- }
- // check that the home setting is actually a directory that exists
- File homedir = new File(home);
- if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
- throw new IOException("Hadoop home directory " + homedir
- + " does not exist, is not a directory, or is not an absolute path.");
- }
- home = homedir.getCanonicalPath();
- } catch (IOException ioe) {
- if (LOG.isDebugEnabled()) {
- LOG.debug("Failed to detect a valid hadoop home directory", ioe);
- }
- home = null;
- }
- //固定本机的hadoop地址
- home="D:\\hadoop-2.2.0";
- return home;
- }
private static String checkHadoopHome() {
// first check the Dflag hadoop.home.dir with JVM scope
//System.setProperty("hadoop.home.dir", "...");
String home = System.getProperty("hadoop.home.dir");
// fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
}
// check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
}
//固定本机的hadoop地址
home="D:\\hadoop-2.2.0";
return home;
}
第二个异常，Could not locate executable D:\Hadoop\tar\hadoop-2.2.0\hadoop-2.2.0\bin\winutils.exe in the Hadoop binaries. 找不到win上的执行程序，可以去https://github.com/srccodes/hadoop-common-2.2.0-bin下载bin包，覆盖本机的hadoop跟目录下的bin包即可
第三个异常：
Java代码
- Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://192.168.130.54:19000/user/hmail/output/part-00000, expected: file:///
- at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
- at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
- at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:357)
- at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
- at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
- at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
- at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
- at com.netease.hadoop.HDFSCatWithAPI.main(HDFSCatWithAPI.java:23)
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://192.168.130.54:19000/user/hmail/output/part-00000, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:357)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
at com.netease.hadoop.HDFSCatWithAPI.main(HDFSCatWithAPI.java:23)
出现这个异常，一般是HDFS的路径写的有问题，解决办法，拷贝集群上的core-site.xml和hdfs-site.xml文件，放在eclipse的src根目录下即可。
第四个异常：
Java代码
- Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
出现这个异常，一般是由于HADOOP_HOME的环境变量配置的有问题，在这里散仙特别说明一下，如果想在Win上的eclipse中成功调试Hadoop2.2，就需要在本机的环境变量上，添加如下的环境变量：
（1）在系统变量中，新建HADOOP_HOME变量，属性值为D:\hadoop-2.2.0.也就是本机对应的hadoop目录
(2)在系统变量的Path里，追加%HADOOP_HOME%/bin即可
以上的问题，是散仙在测试遇到的，经过对症下药，我们的eclipse终于可以成功的调试MR程序了，散仙这里的Hellow World源码如下：
Java代码
- package com.qin.wordcount;
- import java.io.IOException;
- import org.apache.hadoop.fs.FileSystem;
- import org.apache.hadoop.fs.Path;
- import org.apache.hadoop.io.IntWritable;
- import org.apache.hadoop.io.LongWritable;
- import org.apache.hadoop.io.Text;
- import org.apache.hadoop.mapred.JobConf;
- import org.apache.hadoop.mapreduce.Job;
- import org.apache.hadoop.mapreduce.Mapper;
- import org.apache.hadoop.mapreduce.Reducer;
- import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
- import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
- import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
- import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
- /***
- *
- * Hadoop2.2.0测试
- * 放WordCount的例子
- *
- * @author qindongliang
- *
- * hadoop技术交流群： 376932160
- *
- *
- * */
- public class MyWordCount {
- /**
- * Mapper
- *
- * **/
- private static class WMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
- private IntWritable count=new IntWritable(1);
- private Text text=new Text();
- @Override
- protected void map(LongWritable key, Text value,Context context)
- throws IOException, InterruptedException {
- String values[]=value.toString().split("#");
- //System.out.println(values[0]+"========"+values[1]);
- count.set(Integer.parseInt(values[1]));
- text.set(values[0]);
- context.write(text,count);
- }
- }
- /**
- * Reducer
- *
- * **/
- private static class WReducer extends Reducer<Text, IntWritable, Text, Text>{
- private Text t=new Text();
- @Override
- protected void reduce(Text key, Iterable<IntWritable> value,Context context)
- throws IOException, InterruptedException {
- int count=0;
- for(IntWritable i:value){
- count+=i.get();
- }
- t.set(count+"");
- context.write(key,t);
- }
- }
- /**
- * 改动一
- * (1)shell源码里添加checkHadoopHome的路径
- * (2)974行，FileUtils里面
- * **/
- public static void main(String[] args) throws Exception{
- // String path1=System.getenv("HADOOP_HOME");
- // System.out.println(path1);
- // System.exit(0);
- JobConf conf=new JobConf(MyWordCount.class);
- //Configuration conf=new Configuration();
- //conf.set("mapred.job.tracker","192.168.75.130:9001");
- //读取person中的数据字段
- // conf.setJar("tt.jar");
- //注意这行代码放在最前面，进行初始化，否则会报
- /**Job任务**/
- Job job=new Job(conf, "testwordcount");
- job.setJarByClass(MyWordCount.class);
- System.out.println("模式： "+conf.get("mapred.job.tracker"));;
- // job.setCombinerClass(PCombine.class);
- // job.setNumReduceTasks(3);//设置为3
- job.setMapperClass(WMapper.class);
- job.setReducerClass(WReducer.class);
- job.setInputFormatClass(TextInputFormat.class);
- job.setOutputFormatClass(TextOutputFormat.class);
- job.setMapOutputKeyClass(Text.class);
- job.setMapOutputValueClass(IntWritable.class);
- job.setOutputKeyClass(Text.class);
- job.setOutputValueClass(Text.class);
- String path="hdfs://192.168.46.28:9000/qin/output";
- FileSystem fs=FileSystem.get(conf);
- Path p=new Path(path);
- if(fs.exists(p)){
- fs.delete(p, true);
- System.out.println("输出路径存在，已删除！");
- }
- FileInputFormat.setInputPaths(job, "hdfs://192.168.46.28:9000/qin/input");
- FileOutputFormat.setOutputPath(job,p );
- System.exit(job.waitForCompletion(true) ? 0 : 1);
- }
- }
package com.qin.wordcount;
import java.io.IOException;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
/***
*
* Hadoop2.2.0测试
* 放WordCount的例子
*
* @author qindongliang
*
* hadoop技术交流群： 376932160
*
*
* */
public class MyWordCount {
/**
* Mapper
*
* **/
private static class WMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
private IntWritable count=new IntWritable(1);
private Text text=new Text();
@Override
protected void map(LongWritable key, Text value,Context context)
throws IOException, InterruptedException {
String values[]=value.toString().split("#");
//System.out.println(values[0]+"========"+values[1]);
count.set(Integer.parseInt(values[1]));
text.set(values[0]);
context.write(text,count);
}
}
/**
* Reducer
*
* **/
private static class WReducer extends Reducer<Text, IntWritable, Text, Text>{
private Text t=new Text();
@Override
protected void reduce(Text key, Iterable<IntWritable> value,Context context)
throws IOException, InterruptedException {
int count=0;
for(IntWritable i:value){
count+=i.get();
}
t.set(count+"");
context.write(key,t);
}
}
/**
* 改动一
* (1)shell源码里添加checkHadoopHome的路径
* (2)974行，FileUtils里面
* **/
public static void main(String[] args) throws Exception{
// String path1=System.getenv("HADOOP_HOME");
// System.out.println(path1);
// System.exit(0);
JobConf conf=new JobConf(MyWordCount.class);
//Configuration conf=new Configuration();
//conf.set("mapred.job.tracker","192.168.75.130:9001");
//读取person中的数据字段
// conf.setJar("tt.jar");
//注意这行代码放在最前面，进行初始化，否则会报
/**Job任务**/
Job job=new Job(conf, "testwordcount");
job.setJarByClass(MyWordCount.class);
System.out.println("模式： "+conf.get("mapred.job.tracker"));;
// job.setCombinerClass(PCombine.class);
// job.setNumReduceTasks(3);//设置为3
job.setMapperClass(WMapper.class);
job.setReducerClass(WReducer.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
String path="hdfs://192.168.46.28:9000/qin/output";
FileSystem fs=FileSystem.get(conf);
Path p=new Path(path);
if(fs.exists(p)){
fs.delete(p, true);
System.out.println("输出路径存在，已删除！");
}
FileInputFormat.setInputPaths(job, "hdfs://192.168.46.28:9000/qin/input");
FileOutputFormat.setOutputPath(job,p );
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
控制台，打印日志如下：
Java代码
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
- 模式： local
- 输出路径存在，已删除！
- INFO - Configuration.warnOnceIfDeprecated(840) | session.id is deprecated. Instead, use dfs.metrics.session-id
- INFO - JvmMetrics.init(76) | Initializing JVM Metrics with processName=JobTracker, sessionId=
- WARN - JobSubmitter.copyAndConfigureFiles(149) | Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- WARN - JobSubmitter.copyAndConfigureFiles(258) | No job jar file set. User classes may not be found. See Job or Job#setJar(String).
- INFO - FileInputFormat.listStatus(287) | Total input paths to process : 1
- INFO - JobSubmitter.submitJobInternal(394) | number of splits:1
- INFO - Configuration.warnOnceIfDeprecated(840) | user.name is deprecated. Instead, use mapreduce.job.user.name
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.job.name is deprecated. Instead, use mapreduce.job.name
- INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
- INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
- INFO - JobSubmitter.printTokens(477) | Submitting tokens for job: job_local1181216011_0001
- WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/staging/qindongliang1181216011/.staging/job_local1181216011_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
- WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/staging/qindongliang1181216011/.staging/job_local1181216011_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
- WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/local/localRunner/qindongliang/job_local1181216011_0001/job_local1181216011_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
- WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/local/localRunner/qindongliang/job_local1181216011_0001/job_local1181216011_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
- INFO - Job.submit(1272) | The url to track the job: http://localhost:8080/
- INFO - Job.monitorAndPrintJob(1317) | Running job: job_local1181216011_0001
- INFO - LocalJobRunner$Job.createOutputCommitter(323) | OutputCommitter set in config null
- INFO - LocalJobRunner$Job.createOutputCommitter(341) | OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
- INFO - LocalJobRunner$Job.run(389) | Waiting for map tasks
- INFO - LocalJobRunner$Job$MapTaskRunnable.run(216) | Starting task: attempt_local1181216011_0001_m_000000_0
- INFO - ProcfsBasedProcessTree.isAvailable(129) | ProcfsBasedProcessTree currently is supported only on Linux.
- INFO - Task.initialize(581) | Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@39550640
- INFO - MapTask.runNewMapper(732) | Processing split: hdfs://192.168.46.28:9000/qin/input/test.txt:0+38
- INFO - MapTask.createSortingCollector(387) | Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
- INFO - MapTask$MapOutputBuffer.setEquator(1183) | (EQUATOR) 0 kvi 26214396(104857584)
- INFO - MapTask$MapOutputBuffer.init(975) | mapreduce.task.io.sort.mb: 100
- INFO - MapTask$MapOutputBuffer.init(976) | soft limit at 83886080
- INFO - MapTask$MapOutputBuffer.init(977) | bufstart = 0; bufvoid = 104857600
- INFO - MapTask$MapOutputBuffer.init(978) | kvstart = 26214396; length = 6553600
- INFO - LocalJobRunner$Job.statusUpdate(513) |
- INFO - MapTask$MapOutputBuffer.flush(1440) | Starting flush of map output
- INFO - MapTask$MapOutputBuffer.flush(1459) | Spilling map output
- INFO - MapTask$MapOutputBuffer.flush(1460) | bufstart = 0; bufend = 44; bufvoid = 104857600
- INFO - MapTask$MapOutputBuffer.flush(1462) | kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
- INFO - MapTask$MapOutputBuffer.sortAndSpill(1648) | Finished spill 0
- INFO - Task.done(995) | Task:attempt_local1181216011_0001_m_000000_0 is done. And is in the process of committing
- INFO - LocalJobRunner$Job.statusUpdate(513) | map
- INFO - Task.sendDone(1115) | Task 'attempt_local1181216011_0001_m_000000_0' done.
- INFO - LocalJobRunner$Job$MapTaskRunnable.run(241) | Finishing task: attempt_local1181216011_0001_m_000000_0
- INFO - LocalJobRunner$Job.run(397) | Map task executor complete.
- INFO - ProcfsBasedProcessTree.isAvailable(129) | ProcfsBasedProcessTree currently is supported only on Linux.
- INFO - Task.initialize(581) | Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@68843e7b
- INFO - Merger$MergeQueue.merge(568) | Merging 1 sorted segments
- INFO - Merger$MergeQueue.merge(667) | Down to the last merge-pass, with 1 segments left of total size: 45 bytes
- INFO - LocalJobRunner$Job.statusUpdate(513) |
- INFO - Configuration.warnOnceIfDeprecated(840) | mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
- INFO - Task.done(995) | Task:attempt_local1181216011_0001_r_000000_0 is done. And is in the process of committing
- INFO - LocalJobRunner$Job.statusUpdate(513) |
- INFO - Task.commit(1156) | Task attempt_local1181216011_0001_r_000000_0 is allowed to commit now
- INFO - FileOutputCommitter.commitTask(439) | Saved output of task 'attempt_local1181216011_0001_r_000000_0' to hdfs://192.168.46.28:9000/qin/output/_temporary/0/task_local1181216011_0001_r_000000
- INFO - LocalJobRunner$Job.statusUpdate(513) | reduce > reduce
- INFO - Task.sendDone(1115) | Task 'attempt_local1181216011_0001_r_000000_0' done.
- INFO - Job.monitorAndPrintJob(1338) | Job job_local1181216011_0001 running in uber mode : false
- INFO - Job.monitorAndPrintJob(1345) | map 100% reduce 100%
- INFO - Job.monitorAndPrintJob(1356) | Job job_local1181216011_0001 completed successfully
- INFO - Job.monitorAndPrintJob(1363) | Counters: 32
- File System Counters
- FILE: Number of bytes read=372
- FILE: Number of bytes written=382174
- FILE: Number of read operations=0
- FILE: Number of large read operations=0
- FILE: Number of write operations=0
- HDFS: Number of bytes read=76
- HDFS: Number of bytes written=27
- HDFS: Number of read operations=17
- HDFS: Number of large read operations=0
- HDFS: Number of write operations=6
- Map-Reduce Framework
- Map input records=4
- Map output records=4
- Map output bytes=44
- Map output materialized bytes=58
- Input split bytes=109
- Combine input records=0
- Combine output records=0
- Reduce input groups=3
- Reduce shuffle bytes=0
- Reduce input records=4
- Reduce output records=3
- Spilled Records=8
- Shuffled Maps =0
- Failed Shuffles=0
- Merged Map outputs=0
- GC time elapsed (ms)=0
- CPU time spent (ms)=0
- Physical memory (bytes) snapshot=0
- Virtual memory (bytes) snapshot=0
- Total committed heap usage (bytes)=532938752
- File Input Format Counters
- Bytes Read=38
- File Output Format Counters
- Bytes Written=27
INFO - Configuration.warnOnceIfDeprecated(840) | mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 模式： local 输出路径存在，已删除！ INFO - Configuration.warnOnceIfDeprecated(840) | session.id is deprecated. Instead, use dfs.metrics.session-id INFO - JvmMetrics.init(76) | Initializing JVM Metrics with processName=JobTracker, sessionId= WARN - JobSubmitter.copyAndConfigureFiles(149) | Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. WARN - JobSubmitter.copyAndConfigureFiles(258) | No job jar file set. User classes may not be found. See Job or Job#setJar(String). INFO - FileInputFormat.listStatus(287) | Total input paths to process : 1 INFO - JobSubmitter.submitJobInternal(394) | number of splits:1 INFO - Configuration.warnOnceIfDeprecated(840) | user.name is deprecated. Instead, use mapreduce.job.user.name INFO - Configuration.warnOnceIfDeprecated(840) | mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class INFO - Configuration.warnOnceIfDeprecated(840) | mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class INFO - Configuration.warnOnceIfDeprecated(840) | mapred.job.name is deprecated. Instead, use mapreduce.job.name INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class INFO - Configuration.warnOnceIfDeprecated(840) | mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir INFO - Configuration.warnOnceIfDeprecated(840) | mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir INFO - Configuration.warnOnceIfDeprecated(840) | mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class INFO - Configuration.warnOnceIfDeprecated(840) | mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps INFO - Configuration.warnOnceIfDeprecated(840) | mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class INFO - Configuration.warnOnceIfDeprecated(840) | mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class INFO - Configuration.warnOnceIfDeprecated(840) | mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir INFO - JobSubmitter.printTokens(477) | Submitting tokens for job: job_local1181216011_0001 WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/staging/qindongliang1181216011/.staging/job_local1181216011_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/staging/qindongliang1181216011/.staging/job_local1181216011_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/local/localRunner/qindongliang/job_local1181216011_0001/job_local1181216011_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. WARN - Configuration.loadProperty(2172) | file:/root/hadoop/tmp/mapred/local/localRunner/qindongliang/job_local1181216011_0001/job_local1181216011_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. INFO - Job.submit(1272) | The url to track the job: http://localhost:8080/ INFO - Job.monitorAndPrintJob(1317) | Running job: job_local1181216011_0001 INFO - LocalJobRunner$Job.createOutputCommitter(323) | OutputCommitter set in config null INFO - LocalJobRunner$Job.createOutputCommitter(341) | OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter INFO - LocalJobRunner$Job.run(389) | Waiting for map tasks INFO - LocalJobRunner$Job$MapTaskRunnable.run(216) | Starting task: attempt_local1181216011_0001_m_000000_0 INFO - ProcfsBasedProcessTree.isAvailable(129) | ProcfsBasedProcessTree currently is supported only on Linux. INFO - Task.initialize(581) | Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@39550640 INFO - MapTask.runNewMapper(732) | Processing split: hdfs://192.168.46.28:9000/qin/input/test.txt:0+38 INFO - MapTask.createSortingCollector(387) | Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer INFO - MapTask$MapOutputBuffer.setEquator(1183) | (EQUATOR) 0 kvi 26214396(104857584) INFO - MapTask$MapOutputBuffer.init(975) | mapreduce.task.io.sort.mb: 100 INFO - MapTask$MapOutputBuffer.init(976) | soft limit at 83886080 INFO - MapTask$MapOutputBuffer.init(977) | bufstart = 0; bufvoid = 104857600 INFO - MapTask$MapOutputBuffer.init(978) | kvstart = 26214396; length = 6553600 INFO - LocalJobRunner$Job.statusUpdate(513) | INFO - MapTask$MapOutputBuffer.flush(1440) | Starting flush of map output INFO - MapTask$MapOutputBuffer.flush(1459) | Spilling map output INFO - MapTask$MapOutputBuffer.flush(1460) | bufstart = 0; bufend = 44; bufvoid = 104857600 INFO - MapTask$MapOutputBuffer.flush(1462) | kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600 INFO - MapTask$MapOutputBuffer.sortAndSpill(1648) | Finished spill 0 INFO - Task.done(995) | Task:attempt_local1181216011_0001_m_000000_0 is done. And is in the process of committing INFO - LocalJobRunner$Job.statusUpdate(513) | map INFO - Task.sendDone(1115) | Task 'attempt_local1181216011_0001_m_000000_0' done. INFO - LocalJobRunner$Job$MapTaskRunnable.run(241) | Finishing task: attempt_local1181216011_0001_m_000000_0 INFO - LocalJobRunner$Job.run(397) | Map task executor complete. INFO - ProcfsBasedProcessTree.isAvailable(129) | ProcfsBasedProcessTree currently is supported only on Linux. INFO - Task.initialize(581) | Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@68843e7b INFO - Merger$MergeQueue.merge(568) | Merging 1 sorted segments INFO - Merger$MergeQueue.merge(667) | Down to the last merge-pass, with 1 segments left of total size: 45 bytes INFO - LocalJobRunner$Job.statusUpdate(513) | INFO - Configuration.warnOnceIfDeprecated(840) | mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords INFO - Task.done(995) | Task:attempt_local1181216011_0001_r_000000_0 is done. And is in the process of committing INFO - LocalJobRunner$Job.statusUpdate(513) | INFO - Task.commit(1156) | Task attempt_local1181216011_0001_r_000000_0 is allowed to commit now INFO - FileOutputCommitter.commitTask(439) | Saved output of task 'attempt_local1181216011_0001_r_000000_0' to hdfs://192.168.46.28:9000/qin/output/_temporary/0/task_local1181216011_0001_r_000000 INFO - LocalJobRunner$Job.statusUpdate(513) | reduce > reduce INFO - Task.sendDone(1115) | Task 'attempt_local1181216011_0001_r_000000_0' done. INFO - Job.monitorAndPrintJob(1338) | Job job_local1181216011_0001 running in uber mode : false INFO - Job.monitorAndPrintJob(1345) | map 100% reduce 100% INFO - Job.monitorAndPrintJob(1356) | Job job_local1181216011_0001 completed successfully INFO - Job.monitorAndPrintJob(1363) | Counters: 32 File System Counters FILE: Number of bytes read=372 FILE: Number of bytes written=382174 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=76 HDFS: Number of bytes written=27 HDFS: Number of read operations=17 HDFS: Number of large read operations=0 HDFS: Number of write operations=6 Map-Reduce Framework Map input records=4 Map output records=4 Map output bytes=44 Map output materialized bytes=58 Input split bytes=109 Combine input records=0 Combine output records=0 Reduce input groups=3 Reduce shuffle bytes=0 Reduce input records=4 Reduce output records=3 Spilled Records=8 Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=532938752 File Input Format Counters Bytes Read=38 File Output Format Counters Bytes Written=27
输入的测试数据如下：
Java代码
- 中国#1
- 美国#2
- 英国#3
- 中国#2
中国#1 美国#2 英国#3 中国#2
输出的结果如下：
Java代码
- 中国 3
- 美国 2
- 英国 3
中国 3 美国 2 英国 3
至此，我们已经成功的在eclipse里远程调试hadoop成功，调试时，注意散仙，在上文提出的几个问题，如果遇到时，按照对应的方法解决即可。
相关推荐
"eclipse调试java的10个技巧" Eclipse调试Java的10个技巧是Java开发者们在使用Eclipse进行Java开发时的一些高级调试技巧。这些技巧可以帮助开发者更好地调试Java程序，提高开发效率和质量。 1. 条件断点 条件断点...
【长青说安卓】系列专题(七)：Eclipse王者归来——如何使用Eclipse调试Android源码 在Android开发领域，Eclipse曾是开发者的主要工具，尤其在早期，其强大的集成开发环境为开发者提供了便捷的代码编辑、构建和调试...
### Eclipse调试Java的十个高效技巧 #### 一、概述 在开发Java应用程序的过程中，调试是一项不可或缺的任务。良好的调试技能能够帮助开发者快速定位问题并解决它们。Eclipse是一款广泛使用的Java集成开发环境(IDE)...
在文件"1272237.html"和"1272237_files"中，可能包含有关如何使用Eclipse调试PHP的更具体步骤或示例代码。阅读这些文件将加深你对Eclipse调试PHP过程的理解，并能帮助你在实际开发中更好地应用这些技巧。 总的来说...
本教程“Eclipse调试方法入门”旨在为初学者提供一套全面且易懂的调试指南，帮助他们理解和掌握在Eclipse中进行程序调试的关键技能。 首先，我们要理解调试的基本概念。调试是识别和修复程序中错误的过程，通过它...
本文将详细介绍Eclipse调试中的几个常用技巧，包括条件断点、变量断点、方法断点、改变变量值、重新调试、远程调试以及异常断点。 1、条件断点：条件断点允许我们在代码执行时设置特定条件，只有当条件满足时，程序...
### Eclipse调试问题详解 在开发过程中，遇到诸如“eclipse调试问题”这样的错误往往令人头疼。根据提供的代码片段，我们主要关注的是与Eclipse IDE相关的Java编程中的异常处理和调试技巧，尤其是`...
总的来说，JavaScript Eclipse调试插件是提高JavaScript开发效率和质量的关键工具，通过它，开发者可以更好地理解代码运行状态，快速定位和解决问题，从而提升开发工作的专业性和生产力。对于任何使用Eclipse进行...
【Eclipse调试常用技巧】 Eclipse作为一款强大的Java开发集成环境，其调试功能十分强大，对于初学者或者未充分利用其高级特性的开发者来说，掌握这些技巧可以极大地提高工作效率。以下是一些Eclipse调试中的关键点...
尽管图像内容无法直接查看，但是根据标题和描述，我们可以提取以下Eclipse调试的常用技巧： 1. 启动Debug模式：在Eclipse中，可以通过点击工具栏上的“Debug”图标或者右键点击Java文件选择“Debug As”来启动debug...
【Eclipse调试常用技巧】 Eclipse作为一款强大的Java开发集成环境，其调试功能十分强大，对于初学者或者未充分利用这些工具的开发者来说，掌握这些技巧能够显著提高代码调试的效率和准确性。以下是一些Eclipse调试...
Eclipse是一个集成开发环境（IDE），广泛用于Java语言的开发，同时也支持C、...Eclipse调试器的高级功能还包括条件断点、异常断点、监视点等，这些都可以帮助开发者更精细地控制调试过程，快速定位和修复程序中的错误。
【Eclipse调试Bug的七种常用技巧】 在软件开发过程中，调试是不可或缺的一部分，尤其是在使用Eclipse这样的集成开发环境（IDE）时。Eclipse提供了多种强大的调试工具和技术，帮助开发者快速定位并修复问题。以下是...
### Node.js 连接 Eclipse 调试指南 在当今快速发展的软件开发领域，Node.js 作为一种流行的服务器端 JavaScript 运行环境，被广泛应用于构建高效、可扩展的网络应用程序。对于开发者而言，掌握如何有效地调试 Node...
以下将详细介绍标题提及的Eclipse调试的7种方法： 1. **条件断点**：普通断点是代码执行到某行时停止的基础，而条件断点则允许我们设置一个条件，只有当条件满足时，程序才会在该断点处暂停。这有助于减少不必要的...
### Linux下Eclipse调试C语言简介及环境配置 #### 一、引言 随着软件开发技术的不断发展，高效的集成开发环境(IDE)成为了提高开发效率的关键。对于C语言开发者而言，在Linux环境下寻找合适的IDE尤为重要。本文将...
"Eclipse调试方法入门"是学习Eclipse不可或缺的部分，它讲解了如何在Eclipse中设置断点、单步执行、查看变量值、跟踪调用堆栈等调试技巧。这对于查找和修复代码中的错误至关重要，也是提升开发效率的有效手段。 ...
7. **Eclipse调试优势**： 使用Eclipse的图形化调试界面相对于命令行工具gdb来说，通常更加直观和方便。它提供了丰富的调试工具，如断点、变量观察窗口、调用堆栈查看等，有助于理解代码执行流程和调试复杂问题。 ...
在本文中，我们将深入探讨Eclipse IDE中的调试技巧，这对于任何Java开发者来说都是至关重要的技能。Eclipse是一个强大的集成开发环境，其调试功能强大且多样，可以帮助开发者高效地定位并解决问题。 首先，我们来看...
### Eclipse调试小技巧详解 #### 一、引言 对于初入职场的软件开发者来说，掌握高效的调试技巧是非常重要的。本文旨在帮助那些刚从学校步入社会的开发人员以及那些尚未充分利用Eclipse高级调试功能的朋友，更好地...