Lucene多线程操作实现

Lucene多线程操作实现<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" />

对于并发，Lucene 遵循以下规则：

1. 允许任意多的读操作并发，即任意数量用户可同时对同一索引做检索操作。
2. 即便正在进行索引修改操作(索引优化、添加文档、删除文档)，依然允许任意多的检索操作并发执行。
3. 不允许并发修改操作，也就是说同一时间只允许一个索引修改操作。

Lucene内部已经对多线程安全进行了处理，很多操作都使用了 lock 进行多线程同步锁定。只要遵循一定的规则，就可以在多线程环境下安全运行 Lucene。
方案一：
建议：

1. Directotry、Analyzer 都是多线程安全类型，只需建立一个 Singleton 对象即可。
2. 所有线程使用同一个 IndexModifier 对象进行索引修改操作。
3. IndexWriter/IndexReader/IndexModifier/IndexSearcher 最好使用同一个 Directory 对象，否则多线程并发读写时可能引发 FileNotFoundException。

IndexModifier 对象封装了 IndexWriter 和 IndexReader 的常用操作，其内部实现了多线程同步锁定。使用 IndexModifier 可避免同时使用 IndexWriter 和 IndexReader 时需要在多个对象之间进行同步的麻烦。等所有修改操作完成后，记住调用 Close() 方法关闭相关资源。并不是每次操作都需要调用 Optimize()，可以依据特定情况，定期执行优化操作。

--------

以下演示代码简单封装了一个 IndexModifier Signleton 类型，确保多线程使用同一个对象，且只能由最后一个多线程调用 Close 方法关闭。
代码不完善，仅供参考！需要做些修改才能应用于实际项目。

//索引修改器的获取和关闭

import java.io.File;

import java.io.IOException;

import java.io.StringReader;

import java.util.ArrayList;

import java.util.HashMap;

import java.util.Map;

import org.apache.lucene.analysis.Analyzer;

import org.apache.lucene.analysis.standard.StandardAnalyzer;

import org.apache.lucene.index.CorruptIndexException;

import org.apache.lucene.index.IndexModifier;

import org.apache.lucene.store.Directory;

import org.apache.lucene.store.LockObtainFailedException;

import org.apache.lucene.store.RAMDirectory;

public class MyIndexModifier {

private static Analyzer analyzer = new StandardAnalyzer();

private static IndexModifier modifier;

private static ArrayList<Thread> threadList = new ArrayList<Thread>();

private MyIndexModifier() { }

static final File INDEX_DIR = new File("D:/docindex");

public static IndexModifier GetInstance()

{

synchronized (threadList)

{

if (modifier == null)

{

try {

modifier = new IndexModifier(INDEX_DIR, analyzer, false);

//索引性能测试参数配置

modifier.setMergeFactor(1000);

System.out.println("MergeFactor: " + modifier.getMergeFactor());

System.out.println("MaxBufferedDocs: " + modifier.getMaxBufferedDocs());

} catch (CorruptIndexException e) {

e.printStackTrace();

} catch (LockObtainFailedException e) {

e.printStackTrace();

} catch (IOException e) {

e.printStackTrace();

}

}

if (!threadList.contains(Thread.currentThread()))

threadList.add(Thread.currentThread());

return modifier;

}

}

public static void Close()

{

synchronized (threadList)

{

if (threadList.contains(Thread.currentThread()))

threadList.remove(Thread.currentThread());

if (threadList.size() == 0)

{

try {

if (modifier != null)

{

modifier.close();

modifier = null;

}

} catch (CorruptIndexException e) {

e.printStackTrace();

} catch (IOException e) {

e.printStackTrace();

}

}

}

}

}

//线程处理类

import java.io.IOException;

import java.util.Date;

import org.apache.log4j.LogManager;

import org.apache.log4j.Logger;

import org.apache.lucene.document.Document;

import org.apache.lucene.document.Field;

import org.apache.lucene.index.CorruptIndexException;

import org.apache.lucene.index.IndexModifier;

import org.apache.lucene.index.StaleReaderException;

import org.apache.lucene.store.LockObtainFailedException;

import com.miracle.dm.framework.common.TimestampConverter;

public class TestModifer extends Thread{

private static Logger logger = LogManager.getLogger(TestModifer.class);

@Override

public void run() {

IndexModifier writer = MyIndexModifier.GetInstance();

try {

writer.deleteDocument(0);

} catch (StaleReaderException e1) {

// TODO Auto-generated catch block

e1.printStackTrace();

} catch (CorruptIndexException e1) {

// TODO Auto-generated catch block

e1.printStackTrace();

} catch (LockObtainFailedException e1) {

// TODO Auto-generated catch block

e1.printStackTrace();

} catch (IOException e1) {

// TODO Auto-generated catch block

e1.printStackTrace();

}

for (int x = 0; x < 10; x++)

{

Document doc = new Document();

TimestampConverter converter = new TimestampConverter();

Date date = new Date();

String docDate = converter.timestampToShortStr(date);

doc.add(new Field("docDate", docDate , Field.Store.YES, Field.Index.TOKENIZED));

try {

writer.addDocument(doc);

} catch (CorruptIndexException e) {

// TODO Auto-generated catch block

e.printStackTrace();

} catch (LockObtainFailedException e) {

// TODO Auto-generated catch block

e.printStackTrace();

} catch (IOException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

}

logger.debug(""+ Thread.currentThread()+","+ writer.docCount());

MyIndexModifier.Close(); // 注意不是调用 IndexModifier.Close() ！

}

}
多线程测试代码

import java.io.Console;

import java.io.IOException;

import java.util.Date;

import org.apache.log4j.LogManager;

import org.apache.log4j.Logger;

import org.apache.lucene.document.Document;

import org.apache.lucene.document.Field;

import org.apache.lucene.index.CorruptIndexException;

import org.apache.lucene.index.IndexModifier;

import org.apache.lucene.store.LockObtainFailedException;

import com.miracle.dm.framework.common.TimestampConverter;

public class test {

private static Logger logger = LogManager.getLogger(test.class);

public test(){

}

/**

* @param args

*/

public static void main(String[] args) {

for (int i = 0; i < 100; i++)

{

new TestModifer().start();

}

}

}

注意：使用lucene现在的新版本的朋友一定会发现，现在并不推荐使用。而查看API发现IndexModifier已经被IndexWriter代替。再查看IndexWriter，其中提供了新增，删除，更新索引文档的方法。

这里是自己编码来实现，但是我不知道当几千或更多用户在对索引进行操作，那会不会导致close长时间没有运行，而无法检索到最新的更新索引。希望大家帮我考虑一下是否会存在这方面的问题，如果存在该如何解决？

方案二：利用已有的lucene框架，例如compass

它对lucene实现了实时索引。可基于hibernate，当更新数据库时，系统会自动更新索引。

· Compass将lucene、Spring、Hibernate三者结合
转自：http://wemyss.blogbus.com/logs/8014799.html

1.概述

Compass将lucene、Spring、Hibernate三者的起来，以很低很低的成本快速实现企业应用中的搜索功能。

HomePage: http://www.opensymphony.com/compass/

springside里用了compass来做图书搜索，快速建立的流程如下：

1.用简单的compass annotation把Book对象映射到Lucene。

2.配置compass默认提供的基于Spring MVC的Index Controller 和Search Controller。

3.编写查询结果的显示页面，将controller返回的变量显示出来。

2.Object/Search Engine Mapping的 Annotations配置

使用JDK5 的annotation 来进行OSEM(Object/Search Engine Mapping)比用xml文件按简单许多，下面就是简单的搜索类，可见@SearchableID, @SearchableProperty与@SearchableComponent三个标记，分别代表主键、可搜索的属性与关联的，另一个可搜索的对象，另外Compass要求POJO要有默认构造函数，要实现equals()和hashcode():

详细请点击查看springside中的Product.java , Book.java, Category.java

publicclassProduct{

@SearchableId

privateIntegerid;

privateCategorycategory;

privateStringname;

privateDoubleunitprice;

@SearchableProperty(name="name")

publicStringgetName(){

returnthis.name;

}

@SearchableComponent(refAlias="category")

publicCategorygetCategory(){

returnthis.category;

}

publicDoublegetUnitprice(){

returnthis.unitprice;

}

3. 与spring,hibernate集成配置

3.1 spring配置文件

hiberante中的sessionFactory,transactionManager相比大家也是轻车熟路了.这里还是带过(因为不牵扯稿费的问题吗^_^ ).compass已经对对spring集成做了很好的封装，让我们的使用更加简单,我们可以不为compass编写一行代码,就可以做完搜索引擎的检索.下面是compass在spring中的简明配置. 详情点击查看springside中的applicationContext-lucene.xml ：

<?xml:namespace prefix = v ns = "urn:schemas-microsoft-com:vml" /><shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"><stroke joinstyle="miter"></stroke><formulas><f eqn="if lineDrawn pixelLineWidth 0"></f><f eqn="sum @0 1 0"></f><f eqn="sum 0 0 @1"></f><f eqn="prod @2 1 2"></f><f eqn="prod @3 21600 pixelWidth"></f><f eqn="prod @3 21600 pixelHeight"></f><f eqn="sum @0 0 1"></f><f eqn="prod @6 1 2"></f><f eqn="prod @7 21600 pixelWidth"></f><f eqn="sum @8 21600 0"></f><f eqn="prod @7 21600 pixelHeight"></f><f eqn="sum @10 21600 0"></f></formulas><path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"></path><lock v:ext="edit" aspectratio="t"></lock></shapetype><shape id="_x0000_i1025" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><beans>
<shape id="_x0000_i1026" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><beanid="annotationConfiguration"class="org.compass.annotations.config.CompassAnnotationsConfiguration"></bean>
<shape id="_x0000_i1027" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape>
<shape id="_x0000_i1028" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><beanid="compass"class="org.compass.spring.LocalCompassBean">
<shape id="_x0000_i1029" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><!--anontaition式设置-->
<shape id="_x0000_i1030" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><propertyname="classMappings">
<shape id="_x0000_i1031" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><list>
<shape id="_x0000_i1032" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape><value>org.springside.bookstore.domain.Book</value>
<shape id="_x0000_i1033" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape></list>
<shape id="_x0000_i1034" style="WIDTH: 8.25pt; HEIGHT: 12pt" type="#_x0000_t75" alt=""><imagedata src="file:///C:%5CDOCUME~1%5CUSER%5CLOCALS~1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" o:href="mhtml:file://C:\Documents%20and%20Settings\USER\桌面\本周\Compass将lucene、Spring、Hibernate三者结合%20-%20Wemyss's%20Space%20-%20博客大巴.mht!http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif"></imagedata></shape></property><br