Lucene的基本分词过程

一、Analyzer

       Analyzer类是所有分词器的基类，它是个抽象类，所有的子类必须实现

@Override
protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
     return new TokenStreamComponents;
   }

   

    TokenStreamComponents有两个构造方法

Analyzer.TokenStreamComponents(Tokenizer source)

Analyzer.TokenStreamComponents(Tokenizer source, TokenStream result)

 二、TokenStream

    TokenStream能够产生语汇单元序列的类，有两个类型的子类TokenFilter和Tokenizer

    1.Tokenizer是通过读取字符串创建语汇单元的 

    2.TokenFilter封装另个TokenStream去处理语汇单元，产生新的语汇单元

三、构造一个分析器理解分析链的顺序很重要

        看Lunene In Action书上写的，也不知道是不是这样理解的？

public class StopAnalyzer2 extends Analyzer {

    private Set stopWords;

    public StopAnalyzer2(){
        stopWords=StopAnalyzer.ENGLISH_STOP_WORDS_SET;
    }
    public StopAnalyzer2(String[] stopWords){
        this.stopWords=StopFilter.makeStopSet(Version.LUCENE_42, stopWords);
    }
    @Override
    protected TokenStreamComponents createComponents(String arg0, Reader arg1) {
        //首先转换为小写的流
        Tokenizer t=new LowerCaseTokenizer(Version.LUCENE_42,arg1);
          //转换后用处理去除顿词
           return new TokenStreamComponents(t, new StopFilter(Version.LUCENE_42, t, (CharArraySet) stopWords));

//  TokenFilter tt=new LowerCaseFilter(Version.LUCENE_42, new LetterTokenizer(Version.LUCENE_42, arg1));
//   return new TokenStreamComponents(null,new StopFilter(Version.LUCENE_42, tt, (CharArraySet) stopWords));
    }
//结果看是否能把大写The过滤掉
    public static void main(String[] args) throws IOException {
        AnalyzerUtils.displayTokes(new StopAnalyzer2(), "The I love you");
    }

}

 结果：

 修改后

@Override
protected TokenStreamComponents createComponents(String arg0, Reader arg1) {

    TokenStream filter=new StopFilter(Version.LUCENE_42, new LetterTokenizer(Version.LUCENE_42, arg1), (CharArraySet) stopWords);
       return new TokenStreamComponents(t,new LowerCaseFilter(Version.LUCENE_42, filter));

}

 结果：

 四、对不同分次器做的比较

        书上的Util类这个是在网上找的

public class AnalyzerUtils {
    public static void displayTokes(Analyzer analyzer,String text) throws IOException{
        displayTokes(analyzer.tokenStream("contents", new StringReader(text)));
    }

    public static void displayTokes(TokenStream tokenStream) throws IOException{
         CharTermAttribute termAttribute = tokenStream.addAttribute(CharTermAttribute.class);
         tokenStream.reset();  //此行，不能少，不然会报 java.lang.ArrayIndexOutOfBoundsException

         PositionIncrementAttribute posIncr=tokenStream.addAttribute(PositionIncrementAttribute.class);
         OffsetAttribute o=tokenStream.addAttribute(OffsetAttribute.class);
         TypeAttribute type=tokenStream.addAttribute(TypeAttribute.class);
         while(tokenStream.incrementToken()){
             System.out.print("["+termAttribute.toString()+"  "+o.startOffset()+"->"+o.endOffset()+" "+type.type()+"]");

         }

//       while(tokenStream.incrementToken()){
//           System.out.print("["+termAttribute.toString()+"  "+"]");
//
//       }
    }
}