hadoop常见异常

错误解决：

1.如果在bin/hadoop dfs -put /home/test-in input该过程中出现"can only be replicated to node 0, instead of 1",解决办法是打开http://localhost:50070/dfshealth.jsp查看，确定了livenode数目为你的机器实际数目时，再进行put（注：如果有个别datanode没有启动起来，这是正常的，重新格式化文件系统，然后start-up）。

2.如果put时出现java.io.IOException: Not a file: hdfs://localhost:9000/user/icymary/input/test-in 

解决办法是bin/hadoop dfs -rmr input

bin/hadoop dfs -put /home/test-in input

原因是，当执行了多次put之后，就会在分布式文件系统中生成子目录，删除重新put即可。

3.如果在 bin/hadoop jar hadoop-0.16.0-examples.jar wordcount input output该过程中出现"can only be replicated to node 0, instead of 1"，解决办法是，给磁盘释放更多的空间，当时我的空间只有200M了，运行一直报错，折腾了1天。

4.如果 bin/hadoop jar hadoop-0.16.0-examples.jar wordcount input output过程中

INFO mapred.JobClient: map 0% reduce 0%

且一直卡住，在log日志中也没有出现异样，那么解决办法是，把/etc/hosts里面多余的机器名删掉，即可。

-----------------------

异常分析
在进行上述实践过程中，可能会遇到某种异常情况，大致分析如下：
1、Call to localhost/127.0.0.1:9000 failed on local exception异常
（1）异常描述
可能你会在执行如下命令行的时候出现：
[root@localhost hadoop-0.19.2]# bin/hadoop jar hadoop-0.19.0-examples.jar wordcount input output
出错异常信息如下所示：
10/08/01 19:50:55 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).
10/08/01 19:50:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).
10/08/01 19:50:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).
10/08/01 19:50:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).
10/08/01 19:50:59 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).
10/08/01 19:51:00 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).
10/08/01 19:51:01 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).
10/08/01 19:51:02 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).
10/08/01 19:51:03 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).
10/08/01 19:51:04 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).
java.lang.RuntimeException: java.io.IOException: Call to localhost/127.0.0.1:9000 failed on local exception: Connection refused
        at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:323)
        at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:295)
        at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:268)
        at org.apache.hadoop.examples.WordCount.run(WordCount.java:146)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.io.IOException: Call to localhost/127.0.0.1:9000 failed on local exception: Connection refused
        at org.apache.hadoop.ipc.Client.call(Client.java:699)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
        at $Proxy0.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:104)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:74)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1367)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:56)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1379)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
        at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:319)
        ... 21 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
        at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:685)
        ... 33 more
（2）异常分析
从上述异常信息分析，这句是关键：
Retrying connect to server: localhost/127.0.0.1:9000.
是说在尝试10次连接到“server”时都无法成功，这就说明到server的通信链路是不通的。我们已经在hadoop-site.xml中配置了namenode结点的值，如下所示：
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
所以，很可能namenode进程根本就没有启动，更不必谈要执行任务了。
上述异常，过程是：
格式化了HDFS，但是没有执行bin/start-all.sh，直接启动wordcount任务，就出现上述异常。
所以，应该执行bin/start-all.sh以后再启动wordcount任务。
2、Input path does not exist异常
（1）异常描述
当你在当前hadoop目录下面创建一个input目录，并cp某些文件到里面，开始执行：
[root@localhost hadoop-0.19.2]# bin/hadoop namenode -format
[root@localhost hadoop-0.19.2]# bin/start-all.sh  
这时候，你认为input已经存在，应该可以执行wordcount任务了：
[root@localhost hadoop-0.19.2]# bin/hadoop jar hadoop-0.19.0-examples.jar wordcount input output
结果抛出一堆异常，信息如下：
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/input
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179)
        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:782)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1127)
        at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
上述异常，我模拟的过程是：
[root@localhost hadoop-0.19.2]# bin/hadoop fs -rmr input
Deleted hdfs://localhost:9000/user/root/input
[root@localhost hadoop-0.19.2]# bin/hadoop fs -rmr output
Deleted hdfs://localhost:9000/user/root/output
（2）异常分析
本地的input目录并没有上传到HDFS上，所出现org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/input
只需要执行上传的命令即可：
[root@localhost hadoop-0.19.2]# bin/hadoop fs -put input/ input

评论

1 楼 yingjianxuehun 2013-09-01  

INFO mapred.JobClient: map 0% reduce 0%
请问lz，我在伪分布模式下也遇到此问题，但是我看hosts没有别的主机，还有别的原因么？