heritrix扩展，多线程抓取网页

由于下载速度太慢，打算重载QueueAssignmentPolicy这个类，并重写它的getClassKey（）这个方法，网上的代码有：

 public String getClassKey(CrawlController controller, CandidateURI cauri) {
        String uri = cauri.getUURI().toString();
        long hash = ELFHash(uri);
        String a = Long.toString(hash % 100);
        return a;
    }
    public long ELFHash(String str)
    {
       long hash = 0;
       long x    = 0;
       for(int i = 0; i < str.length(); i++)
       {
          hash = (hash << 4) + str.charAt(i);
          if((x = hash & 0xF0000000L) != 0)
          {
             hash ^= (x >> 24);
             hash &= ~x;
          }
       }
       return (hash & 0x7FFFFFFF);
    }
 

 但是重写后怎么才能用呢？这个问题我研究了一上午。

 这个类的作用是在Frontier为线程提供链接的时候提供链接队列的分配策略（默认是org.archive.crawler.frontier.HostnameQueueAssignmentPolicy这个类来提供）由于一般是选用BdbFrontier，（但是BdbFrontier中没有）-->查看父类WorkQueueFrontier(也没有)-->查看父类AbstractFrontier（发现了

    public final static String ATTR_QUEUE_ASSIGNMENT_POLICY =
        "queue-assignment-policy";

 

并在其构造方法中发现了对此变量引用的足迹

// Read the list of permissible choices from heritrix.properties.
        // Its a list of space- or comma-separated values.
//通过“ ATTR_QUEUE_ASSIGNMENT_POLICY”（queue-assignment-policy）获得此键的属性
        String queueStr = System.getProperty(AbstractFrontier.class.getName() +
                "." + ATTR_QUEUE_ASSIGNMENT_POLICY,
                HostnameQueueAssignmentPolicy.class.getName() + " " +
                IPQueueAssignmentPolicy.class.getName() + " " +
                BucketQueueAssignmentPolicy.class.getName() + " " +
                SurtAuthorityQueueAssignmentPolicy.class.getName() + " " +
                TopmostAssignedSurtQueueAssignmentPolicy.class.getName());
//
        Pattern p = Pattern.compile("\\s*,\\s*|\\s+");
//通过正在表达式切分改String，返回一个String数组
        String [] queues = p.split(queueStr);
        if (queues.length <= 0) {
            throw new RuntimeException("Failed parse of " +
                    " assignment queue policy string: " + queueStr);
        }
//创建一个SimpleType加入到definitionMap中
/*具体方法体为：

  public Type addElementToDefinition(Type type) {
        if (isInitialized()) {
            throw new IllegalStateException(
                    "Elements should only be added to definition in the " +
                    "constructor.");
        }
        if (definitionMap.containsKey(type.getName())) {
            definition.remove(definitionMap.remove(type.getName()));
        }
            
        definition.add(type);
        definitionMap.put(type.getName(), type);
        return type;
    }
*/
        t = addElementToDefinition(new SimpleType(ATTR_QUEUE_ASSIGNMENT_POLICY,
                "Defines how to assign URIs to queues. Can assign by host, " +
                "by ip, and into one of a fixed set of buckets (1k).",
                queues[0], queues));

 

其注释也清楚的说明了

 //Read the list of permissible choices from heritrix.properties

 

查看heritrix.properties

 

#####################################################
# FRONTIER
#####################################################
# List here all queue assignment policies you'd have show as a
# queue-assignment-policy choice in AbstractFrontier derived Frontiers
# (e.g. BdbFrontier).
  org.archive.crawler.frontier.AbstractFrontier.queue-assignment-policy =
  org.archive.crawler.frontier.HostnameQueueAssignmentPolicy
  org.archive.crawler.frontier.IPQueueAssignmentPolicy 
  org.archive.crawler.frontier.BucketQueueAssignmentPolicy 
  org.archive.crawler.frontier.SurtAuthorityQueueAssignmentPolicy
  org.archive.crawler.frontier.TopmostAssignedSurtQueueAssignmentPolicy
org.archive.crawler.frontier.BdbFrontier.level = INFO

 

）进一步看

new SimpleType(ATTR_QUEUE_ASSIGNMENT_POLICY,
                "Defines how to assign URIs to queues. Can assign by host, " +
                "by ip, and into one of a fixed set of buckets (1k).",
                queues[0], queues)

该方法最终继承至attribute并把

ATTR_QUEUE_ASSIGNMENT_POLICY，

queues[0]传入Attribute构造方法中。queues[0]即为org.archive.crawler.frontier.HostnameQueueAssignmentPolicy。

 

由以上论述分析可知：

如果想重写QueueAsignmentPolicy，yournameQueueAsignmentPolicy放到

   String queueStr = System.getProperty(AbstractFrontier.class.getName() +
                "." + ATTR_QUEUE_ASSIGNMENT_POLICY,

yourname

QueueAsignmentPolicy.class.getName(0+""+     
                HostnameQueueAssignmentPolicy.class.getName() + " " +
                IPQueueAssignmentPolicy.class.getName() + " " +
                BucketQueueAssignmentPolicy.class.getName() + " " +
                SurtAuthorityQueueAssignmentPolicy.class.getName() + " " +
                TopmostAssignedSurtQueueAssignmentPolicy.class.getName());

在heritrix.propertits中修改如下

#############################################################################
# FRONTIER
#############################################################################

# List here all queue assignment policies you'd have show as a
# queue-assignment-policy choice in AbstractFrontier derived Frontiers
# (e.g. BdbFrontier).
org.archive.crawler.frontier.AbstractFrontier.queue-assignment-policy = userP.myQueueAssignmentPolicy 
org.archive.crawler.frontier.HostnameQueueAssignmentPolicy 
org.archive.crawler.frontier.IPQueueAssignmentPolicy 
org.archive.crawler.frontier.BucketQueueAssignmentPolicy 
org.archive.crawler.frontier.SurtAuthorityQueueAssignmentPolicy 
org.archive.crawler.frontier.TopmostAssignedSurtQueueAssignmentPolicy
org.archive.crawler.frontier.BdbFrontier.level = INFO

 

思路是这样...