Nutch开源搜索引擎的crawl日志分析及工作目录说明

看了nutch关于crawl的源码后，我将crawl的日志分析了一下，主要是熟悉一下整个下载、分析、索引的过程。nutch在整个过程中都是通过Hadoop的MapReduce来实现的。
可以通过nutch来深入学习Hadoop编程，都是比较横的代码。这一块待以后研究完毕后，blog出来。

crawl通过nutch-default.xml参数来控制运行过程，另外需要修改crawl-urlfilter.txt和nutch-site.xml来配合运行，这一点在之前的文章中都提到过，就不再赘述了。

下面是我用来crawl的命令
> bin/nutch crawl urls -dir crawl10 -depth 10 -threads 10 >& nohup.out

crawl started in: crawl10                               //表明网目录络蜘蛛的名称
rootUrlDir = urls //待下载数据的列表文件或列表
threads = 10 //下载线程为10个
depth = 10 //深度是10层
Injector: starting //注入下载列表
Injector: crawlDb: crawl10/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries. //根据注入的列表生成待下载的地址库
Injector: Merging injected urls into crawl db. //执行merge
Injector: done
Generator: Selecting best-scoring urls due for fetch. //判断网页重要性，决定下载顺序
Generator: starting
Generator: segment: crawl10/segments/20080904102201 //生成下载结果存储的数据段
Generator: filtering: false
Generator: topN: 2147483647 //没有指定topN大小，nutch会取默认值
Generator: Partitioning selected urls by host, for politeness.  //将url下载列表按hadoop的中配置文件slaves中定义的datanode来分配。
Generator: done.
Fetcher: starting
Fetcher: segment: crawl10/segments/20080904102201 //下载指定网页内容到segment中去
Fetcher: done
CrawlDb update: starting //下载完毕后，更新下载数据库，增加新的下载
CrawlDb update: db: crawl10/crawldb
CrawlDb update: segments: [crawl10/segments/20080904102201]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done

//循环执行下载
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawl10/segments/20080904102453
Generator: filtering: false
Generator: topN: 2147483647
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: starting
Fetcher: segment: crawl10/segments/20080904102453
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: crawl10/crawldb
CrawlDb update: segments: [crawl10/segments/20080904102453]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done

...... //一共循环10次，Nutch的局域网模式采用了广度优先策略，把二级页面抓取完成以后，进行三级页面抓取。

LinkDb: starting //进行网页链接关系分析
LinkDb: linkdb: crawl10/linkdb
LinkDb: URL normalize: true //规范化
LinkDb: URL filter: true //根据crawl-urlfilter.txt来过滤
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904102201
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904102453
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904102841
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904104322
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904113511
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904132510
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904153615
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904175052
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904194724
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904211956
LinkDb: done //链接分析完毕
Indexer: starting //开始创建索引
Indexer: linkdb: crawl10/linkdb
Indexer: adding segment: /user/nutch/crawl10/segments/20080904102201
Indexer: adding segment: /user/nutch/crawl10/segments/20080904102453
Indexer: adding segment: /user/nutch/crawl10/segments/20080904102841
Indexer: adding segment: /user/nutch/crawl10/segments/20080904104322
Indexer: adding segment: /user/nutch/crawl10/segments/20080904113511
Indexer: adding segment: /user/nutch/crawl10/segments/20080904132510
Indexer: adding segment: /user/nutch/crawl10/segments/20080904153615
Indexer: adding segment: /user/nutch/crawl10/segments/20080904175052
Indexer: adding segment: /user/nutch/crawl10/segments/20080904194724
Indexer: adding segment: /user/nutch/crawl10/segments/20080904211956
Indexer: done //索引创建完毕
Dedup: starting //风页去重
Dedup: adding indexes in: crawl10/indexes
Dedup: done
merging indexes to: crawl10/index //索引合并
Adding /user/nutch/crawl10/indexes/part-00000
Adding /user/nutch/crawl10/indexes/part-00001
Adding /user/nutch/crawl10/indexes/part-00002
Adding /user/nutch/crawl10/indexes/part-00003
done merging //合并完毕
crawl finished: crawl10 //入口注入、循环下载、链接分析、建立索引、去重、合并


抓取程序自动在用户根目录(/user/nutch)下面建立了crawl10目录，可以看到crawldb,segments,index,indexs,linkdb目录,
1)crawldb目录下面存放下载的URL,以及下载的日期，用来页面更新检查时间。
2)linkdb目录存放URL的关联关系，是下载完成后分析时创建的，通过这个关联关系可以实现类似google的pagerank功能。
3)segments目录存储抓取的页面，下面子目录的个数与获取页面的层数有关系，我指定-depth是10层，这个目录下就有10层。
  里面有6个子目录
  content,下载页面的内容
  crawl_fetch,下载URL的状态内容
  crawl_generate,待下载的URL的集合，在generate任务生成时和下载过程中持续分析出来
  crawl_parse,存放用来更新crawldb的外部链接库
  parse_data,存放每个URL解析出来的外部链接和元数据
  parse_text,存放每个解析过的URL的文本内容
4)index目录存放符合lucene格式的索引目录，是indexs里所有的索引内容合并后的完整内容,看了一下这里的索引文件和用lucenedemo做出来的文件名称都不一样，待进一步研究
5)indexs目录存放每次下载的索引目录，存放part-0000到part-0003

crawl流程和工作目录搞清楚了，大家可以进一步拜读一下nutch源码。进一步搞清楚各个目录下面的文件作用，以及文件格式。待续！

评论

1 楼 dt_fb 2011-09-16  

您好，我想问问您，你有跳过recrawl.sh这个脚本文件么？