提升Heritrix3.1.1的抓取效率

 

Heritrix3.1.1是老外写的爬虫，可配置性非常好，但是有一点不好，老外很懂礼貌，所以这个爬虫也很懂礼貌，爬起来非常的慢，1万多个链接，一天也爬不完。

仔细研究它的源代码和文档，网上的文档非常的少，基本要看源代码。

根据网上的这篇文档https://webarchive.jira.com/wiki/display/Heritrix/Politeness+parameters

这个爬虫对网站的礼貌主要是单线程，对同一网站只有一个线程。。。好吧，这个我们没什么办法了，因为它是单线的。

第二个可以设置的地方是politeness参数，这个线程取一个url，处理完成以后，在处理队列里的下一个url之前，先要睡一小觉(snooze)，而睡多长时间，是由DispositionProcessor这个bean来决定的，里面有好几个参数，基本上是根据前一个uri的抓取需要的时间，乘上delayFactor来决定。。。缺省的设置是5倍，就是说，如果前一个uri花了1秒来获取，那么这个线程就要睡5秒。

处理的代码片断如下：

    /**
     * Update any scheduling structures with the new information in this
     * CrawlURI. Chiefly means make necessary arrangements for no other URIs at
     * the same host to be visited within the appropriate politeness window.
     * 
     * @param curi
     *            The CrawlURI
     * @return millisecond politeness delay
     */
    protected long politenessDelayFor(CrawlURI curi) {
        long durationToWait = 0;
        Map<String,Object> cdata = curi.getData();
        if (cdata.containsKey(A_FETCH_BEGAN_TIME)
                && cdata.containsKey(A_FETCH_COMPLETED_TIME)) {

            long completeTime = curi.getFetchCompletedTime();
            long durationTaken = (completeTime - curi.getFetchBeginTime());
            durationToWait = (long)(getDelayFactor() * durationTaken);

            long minDelay = getMinDelayMs();
            if (minDelay > durationToWait) {
                // wait at least the minimum
                durationToWait = minDelay;
            }

            long maxDelay = getMaxDelayMs();
            if (durationToWait > maxDelay) {
                // wait no more than the maximum
                durationToWait = maxDelay;
            }
            
            long respectThreshold = getRespectCrawlDelayUpToSeconds() * 1000;
            if (durationToWait<respectThreshold) {
                // may need to extend wait
                CrawlServer s = getServerCache().getServerFor(curi.getUURI());
                String ua = curi.getUserAgent();
                if (ua == null) {
                    ua = metadata.getUserAgent();
                }
                Robotstxt rep = s.getRobotstxt();
                if (rep != null) {
                    long crawlDelay = (long)(1000 * rep.getDirectivesFor(ua).getCrawlDelay());
                    crawlDelay = 
                        (crawlDelay > respectThreshold) 
                            ? respectThreshold 
                            : crawlDelay;
                    if (crawlDelay > durationToWait) {
                        // wait at least the directive crawl-delay
                        durationToWait = crawlDelay;
                    }
                }
            }
            
            long now = System.currentTimeMillis();
            int maxBandwidthKB = getMaxPerHostBandwidthUsageKbSec();
            if (maxBandwidthKB > 0) {
                // Enforce bandwidth limit
                ServerCache cache = this.getServerCache();
                CrawlHost host = cache.getHostFor(curi.getUURI());
                long minDurationToWait = host.getEarliestNextURIEmitTime()
                        - now;
                float maxBandwidth = maxBandwidthKB * 1.024F; // kilo factor
                long processedBytes = curi.getContentSize();
                host
                        .setEarliestNextURIEmitTime((long)(processedBytes / maxBandwidth)
                                + now);

                if (minDurationToWait > durationToWait) {
                    durationToWait = minDurationToWait;
                }
            }
        }
        return durationToWait;
    }
 

仔细看一下上面的代码，发现maxDelayMs这个参数可以直接设置为0。。。哈哈，改了一下再试，这个线程就没法睡觉了。。。睡啥啊，干完活想睡多久睡多久

 <bean id="disposition" class="org.archive.crawler.postprocessor.DispositionProcessor">
  <!-- <property name="delayFactor" value="5.0" /> -->
  <!-- <property name="minDelayMs" value="3000" /> -->
  <!-- <property name="respectCrawlDelayUpToSeconds" value="300" /> -->
  <property name="maxDelayMs" value="0" />
  <!-- <property name="maxPerHostBandwidthUsageKbSec" value="0" /> -->
 </bean>

实战结果统计为：

0.57 URIs/sec (0.64 avg); 35 KB/sec (28 avg)

而之前这个值为：

 0.1 URIs/sec (0.07 avg)       6 KB/sec(4 avg)

提升了5倍左右