`

配置完成nutch容易出现的错误

阅读更多

配置完成nutch容易出现的错误

1.1.2   运行crawl报错Job failed

Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
        at org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:439)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:135)
问题解决:

此多为crawl-urlfilter.txt:MY.DOMAIN.NAME的修改不正确



1.1.3   又一个Job failed

Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
        at org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:439)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:135)

问题解决:

1、多为crawl-urlfilter.txt的MY.DOMAIN.NAME修改不正确

2、中断过正在抓取的程序
3、刚集合了庖丁分词

1.1.4   Eclipse中运行nutch:Job failed

Exception in thread "main" java.io.IOException: Job failed!
       at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
      at org.apache.nutch.crawl.Injector.inject(Injector.java:162)
       at org.apache.nutch.crawl.Crawl.main(Crawl.java:115)

问题解决:

此问题是eclipse的java版本设置问题,解决方法:

如原来使用java1.4,需要改为1.6

project-》properties-》java compiler

右 jdk compliance

compiler compliance level:改为6.0



执行bin/nutch crawl urls -dir crawled -depth 3 -thrads 4 -topN 5 >& crawl.log抛如下异常:

Exception in thread "main" java.net.UnknownHostException: unknown host: namenode
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:850)
    at org.apache.hadoop.ipc.Client.call(Client.java:720)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at org.apache.nutch.crawl.Crawl.main(Crawl.java:94)

分享到:
评论

相关推荐

Global site tag (gtag.js) - Google Analytics