Lucene内置很多的分词器工具包，

Lucene内置很多的分词器工具包，几乎涵盖了全球所有的国家和地区，最近散仙，在搞多语言分词的一个处理，主要国家有西班牙，葡萄牙，德语，法语，意大利，其实这些语系都与英语非常类似，都是以空格为分割的语种。


那么首先，探讨下分词器的词形还原和词干提取的对搜索的意义？在这之前，先看下两者的概念：
词形还原（lemmatization），是把一个任何形式的语言词汇还原为一般形式（能表达完整语义），而词干提取

（stemming）是抽取词的词干或词根形式（不一定能够表达完整语义）。词形还原和词干提取是词形规范化的两类
重要方式，都能够达到有效归并词形的目的，二者既有联系也有区别

详细介绍，请参考这篇文章


在电商搜索里，词干的抽取，和单复数的还原比较重要（这里主要针对名词来讲），因为这有关搜索的查准率，和查全率的命中，如果我们的分词器没有对这些词做过处理，会造成什么影响呢？那么请看如下的一个例子？

句子： i have two cats

分词器如果什么都没有做：

这时候我们搜cat，就会无命中结果，而必须搜cats才能命中到一条数据，而事实上cat和cats是同一个东西，只不过单词的形式不一样，这样以来，如果不做处理，我们的查全率和查全率都会下降，会涉及影响到我们的搜索体验，所以stemming这一步，在某些场合的分词中至关重要。

本篇，散仙，会参考源码分析一下，关于德语分词中中如何做的词干提取，先看下德语的分词声明：




Java代码 复制代码 收藏代码
1.List<String> list=new ArrayList<String>(); 
2.list.add("player");//这里面的词，不会被做词干抽取，词形还原 
3.CharArraySet ar=new CharArraySet(Version.LUCENE_43,list , true); 
4.//分词器的第二个参数是禁用词参数，第三个参数是排除不做词形转换，或单复数的词 
5.GermanAnalyzer sa=new GermanAnalyzer(Version.LUCENE_43,null,ar); 



接着，我们具体看下，在德语的分词器中，都经过了哪几部分的过滤处理：



Java代码 复制代码 收藏代码
1. protected TokenStreamComponents createComponents(String fieldName, 
2.     Reader reader) { 
3.  //标准分词器过滤 
4.   final Tokenizer source = new StandardTokenizer(matchVersion, reader); 
5.   TokenStream result = new StandardFilter(matchVersion, source); 
6.//转小写过滤 
7.   result = new LowerCaseFilter(matchVersion, result); 
8.//禁用词过滤 
9.   result = new StopFilter( matchVersion, result, stopwords); 
10.//排除词过滤 
11.   result = new SetKeywordMarkerFilter(result, exclusionSet); 
12.   if (matchVersion.onOrAfter(Version.LUCENE_36)) { 
13.//在lucene3.6以后的版本，采用如下filter过滤 
14.  //规格化，将德语中的特殊字符，映射成英语 
15.     result = new GermanNormalizationFilter(result); 
16.  //stem词干抽取，词性还原 
17.     result = new GermanLightStemFilter(result); 
18.   } else if (matchVersion.onOrAfter(Version.LUCENE_31)) { 
19.//在lucene3.1至3.6的版本中，采用SnowballFilter处理 
20.     result = new SnowballFilter(result, new German2Stemmer()); 
21.   } else { 
22.//在lucene3.1之前的采用兼容的GermanStemFilter处理 
23.     result = new GermanStemFilter(result); 
24.   } 
25.   return new TokenStreamComponents(source, result); 
26. } 



OK，我们从源码中得知，在Lucene4.x中对德语的分词也做了向前和向后兼容，现在我们主要关注在lucene4.x之后的版本如何的词形转换，下面分别看下
     result = new GermanNormalizationFilter(result);
      result = new GermanLightStemFilter(result);
这两个类的功能：




Java代码 复制代码 收藏代码
1.package org.apache.lucene.analysis.de; 
2. 
3./*
4. * Licensed to the Apache Software Foundation (ASF) under one or more
5. * contributor license agreements.  See the NOTICE file distributed with
6. * this work for additional information regarding copyright ownership.
7. * The ASF licenses this file to You under the Apache License, Version 2.0
8. * (the "License"); you may not use this file except in compliance with
9. * the License.  You may obtain a copy of the License at
10. *
11. *     http://www.apache.org/licenses/LICENSE-2.0
12. *
13. * Unless required by applicable law or agreed to in writing, software
14. * distributed under the License is distributed on an "AS IS" BASIS,
15. * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16. * See the License for the specific language governing permissions and
17. * limitations under the License.
18. */ 
19. 
20.import java.io.IOException; 
21. 
22.import org.apache.lucene.analysis.TokenFilter; 
23.import org.apache.lucene.analysis.TokenStream; 
24.import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; 
25.import org.apache.lucene.analysis.util.StemmerUtil; 
26. 
27./**
28. * Normalizes German characters according to the heuristics
29. * of the <a href="http://snowball.tartarus.org/algorithms/german2/stemmer.html">
30. * German2 snowball algorithm</a>.
31. * It allows for the fact that ä, ö and ü are sometimes written as ae, oe and ue.
32. * 
33. *

34. *   <li> 'ß' is replaced by 'ss'
35. *   <li> 'ä', 'ö', 'ü' are replaced by 'a', 'o', 'u', respectively.
36. *   <li> 'ae' and 'oe' are replaced by 'a', and 'o', respectively.
37. *   <li> 'ue' is replaced by 'u', when not following a vowel or q.
38. *

39. * <p>
40. * This is useful if you want this normalization without using
41. * the German2 stemmer, or perhaps no stemming at all.
42. *上面的解释说得很清楚，主要是对德文的一些特殊字母，转换成对应的英文处理
43. *
44. */ 
45.  
46.public final class GermanNormalizationFilter extends TokenFilter { 
47.  // FSM with 3 states: 
48.  private static final int N = 0; /* ordinary state */ 
49.  private static final int V = 1; /* stops 'u' from entering umlaut state */ 
50.  private static final int U = 2; /* umlaut state, allows e-deletion */ 
51. 
52.  private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); 
53.   
54.  public GermanNormalizationFilter(TokenStream input) { 
55.    super(input); 
56.  } 
57. 
58.  @Override 
59.  public boolean incrementToken() throws IOException { 
60.    if (input.incrementToken()) { 
61.      int state = N; 
62.      char buffer[] = termAtt.buffer(); 
63.      int length = termAtt.length(); 
64.      for (int i = 0; i < length; i++) { 
65.        final char c = buffer[i]; 
66.        switch(c) { 
67.          case 'a': 
68.          case 'o': 
69.            state = U; 
70.            break; 
71.          case 'u': 
72.            state = (state == N) ? U : V; 
73.            break; 
74.          case 'e': 
75.            if (state == U) 
76.              length = StemmerUtil.delete(buffer, i--, length); 
77.            state = V; 
78.            break; 
79.          case 'i': 
80.          case 'q': 
81.          case 'y': 
82.            state = V; 
83.            break; 
84.          case 'ä': 
85.            buffer[i] = 'a'; 
86.            state = V; 
87.            break; 
88.          case 'ö': 
89.            buffer[i] = 'o'; 
90.            state = V; 
91.            break; 
92.          case 'ü':  
93.            buffer[i] = 'u'; 
94.            state = V; 
95.            break; 
96.          case 'ß': 
97.            buffer[i++] = 's'; 
98.            buffer = termAtt.resizeBuffer(1+length); 
99.            if (i < length) 
100.              System.arraycopy(buffer, i, buffer, i+1, (length-i)); 
101.            buffer[i] = 's'; 
102.            length++; 
103.            state = N; 
104.            break; 
105.          default: 
106.            state = N; 
107.        } 
108.      } 
109.      termAtt.setLength(length); 
110.      return true; 
111.    } else { 
112.      return false; 
113.    } 
114.  } 
115.} 





Java代码 复制代码 收藏代码
1.package org.apache.lucene.analysis.de; 
2. 
3./*
4. * Licensed to the Apache Software Foundation (ASF) under one or more
5. * contributor license agreements.  See the NOTICE file distributed with
6. * this work for additional information regarding copyright ownership.
7. * The ASF licenses this file to You under the Apache License, Version 2.0
8. * (the "License"); you may not use this file except in compliance with
9. * the License.  You may obtain a copy of the License at
10. *
11. *     http://www.apache.org/licenses/LICENSE-2.0
12. *
13. * Unless required by applicable law or agreed to in writing, software
14. * distributed under the License is distributed on an "AS IS" BASIS,
15. * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16. * See the License for the specific language governing permissions and
17. * limitations under the License.
18. */ 
19. 
20.import java.io.IOException; 
21. 
22.import org.apache.lucene.analysis.TokenFilter; 
23.import org.apache.lucene.analysis.TokenStream; 
24.import org.apache.lucene.analysis.miscellaneous.SetKeywordMarkerFilter; 
25.import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; 
26.import org.apache.lucene.analysis.tokenattributes.KeywordAttribute; 
27. 
28./**
29. * A {@link TokenFilter} that applies {@link GermanLightStemmer} to stem German
30. * words.
31. * <p>
32. * To prevent terms from being stemmed use an instance of
33. * {@link SetKeywordMarkerFilter} or a custom {@link TokenFilter} that sets
34. * the {@link KeywordAttribute} before this {@link TokenStream}.
35. * 
36.
37. *
38. *
39. *这个类，主要做Stemmer（词干提取），而我们主要关注
40. *GermanLightStemmer这个类的作用
41. *
42. *
43. */ 
44.public final class GermanLightStemFilter extends TokenFilter { 
45.  private final GermanLightStemmer stemmer = new GermanLightStemmer(); 
46.  private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); 
47.  private final KeywordAttribute keywordAttr = addAttribute(KeywordAttribute.class); 
48. 
49.  public GermanLightStemFilter(TokenStream input) { 
50.    super(input); 
51.  } 
52.   
53.  @Override 
54.  public boolean incrementToken() throws IOException { 
55.    if (input.incrementToken()) { 
56.      if (!keywordAttr.isKeyword()) { 
57.        final int newlen = stemmer.stem(termAtt.buffer(), termAtt.length()); 
58.        termAtt.setLength(newlen); 
59.      } 
60.      return true; 
61.    } else { 
62.      return false; 
63.    } 
64.  } 
65.} 


下面看下，在GermanLightStemmer中，如何做的词干提取：源码如下：



Java代码 复制代码 收藏代码
1. package org.apache.lucene.analysis.de; 
2. 
3./*
4. * Licensed to the Apache Software Foundation (ASF) under one or more
5. * contributor license agreements.  See the NOTICE file distributed with
6. * this work for additional information regarding copyright ownership.
7. * The ASF licenses this file to You under the Apache License, Version 2.0
8. * (the "License"); you may not use this file except in compliance with
9. * the License.  You may obtain a copy of the License at
10. *
11. *     http://www.apache.org/licenses/LICENSE-2.0
12. *
13. * Unless required by applicable law or agreed to in writing, software
14. * distributed under the License is distributed on an "AS IS" BASIS,
15. * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16. * See the License for the specific language governing permissions and
17. * limitations under the License.
18. */ 
19. 
20./* 
21. * This algorithm is updated based on code located at:
22. * http://members.unine.ch/jacques.savoy/clef/
23. * 
24. * Full copyright for that code follows:
25. */ 
26. 
27./*
28. * Copyright (c) 2005, Jacques Savoy
29. * All rights reserved.
30. *
31. * Redistribution and use in source and binary forms, with or without 
32. * modification, are permitted provided that the following conditions are met:
33. *
34. * Redistributions of source code must retain the above copyright notice, this 
35. * list of conditions and the following disclaimer. Redistributions in binary 
36. * form must reproduce the above copyright notice, this list of conditions and
37. * the following disclaimer in the documentation and/or other materials 
38. * provided with the distribution. Neither the name of the author nor the names 
39. * of its contributors may be used to endorse or promote products derived from 
40. * this software without specific prior written permission.
41. * 
42. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 
43. * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
44. * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 
45. * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 
46. * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
47. * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
48. * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
49. * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
50. * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
51. * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
52. * POSSIBILITY OF SUCH DAMAGE.
53. */ 
54. 
55./**
56. * Light Stemmer for German.
57. * <p>
58. * This stemmer implements the "UniNE" algorithm in:
59. * <i>Light Stemming Approaches for the French, Portuguese, German and Hungarian Languages</i>
60. * Jacques Savoy
61. */ 
62.public class GermanLightStemmer { 
63.   
64.  //处理特殊字符映射 
65.  public int stem(char s[], int len) {    
66.    for (int i = 0; i < len; i++) 
67.      switch(s[i]) { 
68.        case 'ä': 
69.        case 'à': 
70.        case 'á': 
71.        case 'â': s[i] = 'a'; break; 
72.        case 'ö': 
73.        case 'ò': 
74.        case 'ó': 
75.        case 'ô': s[i] = 'o'; break; 
76.        case 'ï': 
77.        case 'ì': 
78.        case 'í': 
79.        case 'î': s[i] = 'i'; break; 
80.        case 'ü':  
81.        case 'ù':  
82.        case 'ú': 
83.        case 'û': s[i] = 'u'; break; 
84.      } 
85.     
86.    len = step1(s, len); 
87.    return step2(s, len); 
88.  } 
89.   
90.   
91.  private boolean stEnding(char ch) { 
92.    switch(ch) { 
93.      case 'b': 
94.      case 'd': 
95.      case 'f': 
96.      case 'g': 
97.      case 'h': 
98.      case 'k': 
99.      case 'l': 
100.      case 'm': 
101.      case 'n': 
102.      case 't': return true; 
103.      default: return false; 
104.    } 
105.  } 
106.  //处理基于以下规则的词干抽取和缩减 
107.  private int step1(char s[], int len) { 
108.    if (len > 5 && s[len-3] == 'e' && s[len-2] == 'r' && s[len-1] == 'n') 
109.      return len - 3; 
110.     
111.    if (len > 4 && s[len-2] == 'e') 
112.      switch(s[len-1]) { 
113.        case 'm': 
114.        case 'n': 
115.        case 'r': 
116.        case 's': return len - 2; 
117.      } 
118.     
119.    if (len > 3 && s[len-1] == 'e') 
120.      return len - 1; 
121.     
122.    if (len > 3 && s[len-1] == 's' && stEnding(s[len-2])) 
123.      return len - 1; 
124.     
125.    return len; 
126.  } 
127.  //处理基于以下规则est，er，en等的词干抽取和缩减 
128.  private int step2(char s[], int len) { 
129.    if (len > 5 && s[len-3] == 'e' && s[len-2] == 's' && s[len-1] == 't') 
130.      return len - 3; 
131.     
132.    if (len > 4 && s[len-2] == 'e' && (s[len-1] == 'r' || s[len-1] == 'n')) 
133.      return len - 2; 
134.     
135.    if (len > 4 && s[len-2] == 's' && s[len-1] == 't' && stEnding(s[len-3])) 
136.      return len - 2; 
137.     
138.    return len; 
139.  } 
140.} 


具体的分析结果如下：



Java代码 复制代码 收藏代码
1.搜索技术交流群：324714439 
2.大数据hadoop交流群：376932160 
3. 
4.0，将一些德语特殊字符，替换成对应的英文表示 
5.1，将所有词干元音还原 a ，o，i，u 
6.ste（2）（按先后顺序，符合以下任意一项，就完成一次校验（return）） 
7.2，单词长度大于5的词，以ern结尾的，直接去掉 
8.3，单词长度大于4的词，以em，en，es，er结尾的，直接去掉 
9.4，单词长度大于3的词，以e结尾的直接去掉 
10.5，单词长度大于3的词，以bs，ds，fs，gs，hs，ks，ls，ms，ns，ts结尾的，直接去掉s 
11.step（3）（按先后顺序，符合以下任意一项，就完成一次校验（return）） 
12.6，单词长度大于5的词，以est结尾的，直接去掉 
13.7，单词长度大于4的词，以er或en结尾的直接去掉 
14.8，单词长度大于4的词，bst，dst，fst，gst，hst，kst，lst，mst，nst，tst，直接去掉后两位字母st 


最后，结合网上资料分析，基于er，en，e，s结尾的是做单复数转换的，其他的几条规则主要是对非名词的单词，做词干抽取。