Lucene的基本分词过程

一、Analyzer

       Analyzer类是所有分词器的基类，它是个抽象类，所有的子类必须实现

@Override
protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
     return new TokenStreamComponents;
   }

   

    TokenStreamComponents有两个构造方法

Analyzer.TokenStreamComponents(Tokenizer source)

Analyzer.TokenStreamComponents(Tokenizer source, TokenStream result)

 二、TokenStream

    TokenStream能够产生语汇单元序列的类，有两个类型的子类TokenFilter和Tokenizer

    1.Tokenizer是通过读取字符串创建语汇单元的 

    2.TokenFilter封装另个TokenStream去处理语汇单元，产生新的语汇单元

三、构造一个分析器理解分析链的顺序很重要

        看Lunene In Action书上写的，也不知道是不是这样理解的？

public class StopAnalyzer2 extends Analyzer {

	private Set stopWords;
	
	public StopAnalyzer2(){
		stopWords=StopAnalyzer.ENGLISH_STOP_WORDS_SET;
	}
	public StopAnalyzer2(String[] stopWords){
		this.stopWords=StopFilter.makeStopSet(Version.LUCENE_42, stopWords);
	}
	@Override
	protected TokenStreamComponents createComponents(String arg0, Reader arg1) {
	    //首先转换为小写的流
		Tokenizer t=new LowerCaseTokenizer(Version.LUCENE_42,arg1);
          //转换后用处理去除顿词
		   return new TokenStreamComponents(t, new StopFilter(Version.LUCENE_42, t, (CharArraySet) stopWords));
		
//  TokenFilter tt=new LowerCaseFilter(Version.LUCENE_42, new LetterTokenizer(Version.LUCENE_42, arg1));
//   return new TokenStreamComponents(null,new StopFilter(Version.LUCENE_42, tt, (CharArraySet) stopWords));
	}
//结果看是否能把大写The过滤掉
	public static void main(String[] args) throws IOException {
		AnalyzerUtils.displayTokes(new StopAnalyzer2(), "The I love you");
	}

}

 结果：

 修改后

	@Override
	protected TokenStreamComponents createComponents(String arg0, Reader arg1) {

		TokenStream filter=new StopFilter(Version.LUCENE_42, new LetterTokenizer(Version.LUCENE_42, arg1), (CharArraySet) stopWords);
		   return new TokenStreamComponents(t,new LowerCaseFilter(Version.LUCENE_42, filter));

	}

 结果：

 四、对不同分次器做的比较

        书上的Util类这个是在网上找的

public class AnalyzerUtils {
    public static void displayTokes(Analyzer analyzer,String text) throws IOException{
    	displayTokes(analyzer.tokenStream("contents", new StringReader(text)));
    }
    
    public static void displayTokes(TokenStream tokenStream) throws IOException{
    	 CharTermAttribute termAttribute = tokenStream.addAttribute(CharTermAttribute.class);
    	 tokenStream.reset();  //此行，不能少，不然会报 java.lang.ArrayIndexOutOfBoundsException
 
   	     PositionIncrementAttribute posIncr=tokenStream.addAttribute(PositionIncrementAttribute.class);
   	     OffsetAttribute o=tokenStream.addAttribute(OffsetAttribute.class);
   	     TypeAttribute type=tokenStream.addAttribute(TypeAttribute.class);
    	 while(tokenStream.incrementToken()){
    		 System.out.print("["+termAttribute.toString()+"  "+o.startOffset()+"->"+o.endOffset()+" "+type.type()+"]");
    		 
    	 }
    
//    	 while(tokenStream.incrementToken()){
//    		 System.out.print("["+termAttribute.toString()+"  "+"]");
//    		 
//    	 }
    }
}