org.archive.crawler.framework.ToeThread

1、controller.getFetchChain().process(curi,this);
1.1、org.archive.crawler.prefetch.Preselector,
1.2、org.archive.crawler.prefetch.PreconditionEnforcer,
1.3、org.archive.modules.fetcher.FetchDNS,
//httpclient
1.4、org.archive.modules.fetcher.FetchHTTP,
1.5、org.archive.modules.extractor.ExtractorHTTP,
1.6、org.archive.modules.extractor.ExtractorHTML,
1.7、org.archive.modules.extractor.ExtractorCSS,
1.8、org.archive.modules.extractor.ExtractorJS,
1.9、org.archive.modules.extractor.ExtractorSWF

2、controller.getFrontier().beginDisposition(curi);
BdbFrontier.beginDisposition(curi)

3、controller.getDispositionChain().process(curi,this);
3.1、org.archive.modules.writer.MirrorWriterProcessor
//getCandidateChain().process(prereq, null);//add seeds
3.2、org.archive.crawler.postprocessor.CandidatesProcessor
     //run isInScope(curi)
     3.2.1、org.archive.crawler.prefetch.CandidateScoper
     3.2.2、org.archive.crawler.prefetch.FrontierPreparer
        // set schedulingDirective
        3.2.2.1、curi.setSchedulingDirective(getSchedulingDirective(curi));
        // set canonicalized version
        3.2.2.2、curi.setCanonicalString(canonicalize(curi));
        // set queue key
        3.2.2.3、curi.setClassKey(getClassKey(curi));
        // set cost
        3.2.2.4、curi.setHolderCost(getCost(curi));
        // set URI precedence
        3.2.2.5、getUriPrecedencePolicy().uriScheduled(curi);
3.3、org.archive.crawler.postprocessor.DispositionProcessor