支持Solr4的IKTokenizerFactory

近日公司开始改造旧有的搜索系统，用到solr4，而且本人关注IK分词器比较多。发现IK 2012对TokenizerFactory不支持，缺乏这个支持的话，有很多自定义配置功能用不上。

故开发此代码，兼容Solr4分词功能

代码测试可用

 

附件附上对IK2012代码的二次编译jar包

 

IKTokenizerFactory

 

package org.wltea.analyzer.henry;

import java.io.IOException;
import java.io.Reader;
import java.util.Map;

import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.util.ResourceLoader;
import org.apache.lucene.analysis.util.ResourceLoaderAware;
import org.apache.lucene.analysis.util.TokenizerFactory;
import org.wltea.analyzer.lucene.IKTokenizer;

/**
 * 主要用于Solr分词使用
 * 
 * @author niko
 * @date 2012-12-25
 * @version
 */
public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoaderAware {
	
	private boolean useSmart;
	
	@Override
	public void init(Map<String, String> args) {
		super.init(args);
		assureMatchVersion();
	}
		
	@Override
	public Tokenizer create(Reader input) {
		IKTokenizer ikTokenizer = new IKTokenizer(input, useSmart);
		return ikTokenizer;
	}

	@Override
	public void inform(ResourceLoader loader) throws IOException {
		useSmart = getBoolean("useSmart", true);
		
	}

}

 schema.xml调用方法：

 <fieldType name="text_ik" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="org.wltea.analyzer.henry.IKTokenizerFactory" useSmart="true"/>
        <!--
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
        -->
        <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
        -->
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="org.wltea.analyzer.henry.IKTokenizerFactory" useSmart="false"/>
        <!--
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
        -->
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

评论

1 楼 snowpeakjava 2014-10-10  

您好，我现在也遇到同样的问题，使用您写的IKTokenizerFactory 类后，配置上去后就找不到节点了。在管理页面上新增节点时有报错。谢谢！