Lucene/Solr Dev 2 : Indexing and Searching Simultaneously

Abstract

   First i give my conclusion: Lucene can index and search simultaneously, what's more, we can only index one document and then we can search this document at once. Sounds cool, but to do like this is necessary or useful? The answer is NO.  To account for the detail reason, we should first delve into Lucene Buffering and Flushing mechanism.

 

Lucene Buffering and Flushing

 

 

As shown in the above figure, when new Documents are add to a Lucene Index, or deletions are pending, they're initially buffered in memory instead of being immediately written to the disk. This is done for performance, to minimize disk IO. Periodcally, this changes are flushed to the index Directory as a new segment. The flush can be triggered by IndexWriter according to the below three criteria:

1. To flush when the buffer has consumed more then a pre-set amount of RAM, we can use the below method to set the buffer size:

IndexWriter writer = getWriter();
writer.setRAMBufferSizeMB(10);

 2. If your IndexWriter called the setMaxBufferedDocs() Method, the flush will be triggered when the specific number of Documents has added,like the below code:

IndexWriter writer = getWriter();
writer.setMaxBufferedDocs(100);

 3. Like the Criteria 2 if your IndexWriter called setMaxBufferedDeleteTerms():

IndexWriter writer = getWriter();
writer.setMaxBufferedDeleteTerms(100);

 In addition, by deffault , IndexWriter flushs only when RAM usage is 16MB.

By now you may be have a question, while the flush has be triggered and the IndexReader can Search this flushed Document? actually, when a flush occurs, the writer creates new segment and deletion files in the Directory. However,these files are neither visible nor usable to a newly opened IndexReader until the writer commits the changes and the reader is reopened. It’s important to understand this difference. Flushing is done to free up memory consumed by buffered changes to the index, whereas committing is done to make all changes(buffered or already flushed) persistent and visible in the index. This means IndexReader always sees the starting state of the index (when IndexWriter was opened), until the writer commits.

Turn to this Articles Topic, if you want to index and search simultaneously, or even if you want to index one document and then search this document at once, your IndexWriter must call commit() method when your IndexWriter only add a Document. To continue explain Lucene index and search simultaneously, we should delve into Lucene index commits.

 

Lucene index commits

 At the begin of this article, we do not need to commit frequently during a indexing, because commit is a costly operation, and doing so frequently will slow down your indexing throughput.

I have made a experiment, indexing 100MB resource File that means adding almost 68768 Dcuments to index directory, and i got the following results

1. if we can not call IndexWriter's commit() Method during the whole indexing process, it only need 25 seconds

 2. if we call commit() method frequemtly,  i called the commit() after every added a Documment to index directory, means i do commit 68768 times during indexing the same 100 MB resource file, and the indexing time is more than 2 hours. it is obvious that do index commit more frequent is not a good choice.

3. if we call commit method while 5000 Documents has added to the index directory, we also indexing 100 MB resouece file, add 68768 Documents. in this condition, we call commit method 13 times, and the final indexing time is  31 seconds.

4. and if we call commit method while 500 Documents has added, the final indexing time is 46 seconds

5. and while 100 Documents has added the indexing time is 115 seconds.

To sum up this results to a table, it might more strightforward:

 

indexing 100MB resource file, adding 68768 documents to index directory
call commit frequency	do commit times	indexing time
never	0	25 seconds
5000 documents has added	13	31 seconds
500 documents has added	137	46 seconds
100 documents has added	687	115 seconds
50 documents has added	1375	210 seconds
1 document has added	68768	more than 2 hours

through the table we can get the conclusion.

 

Conclusion

Lucene indexing and searching simultaneous is possible, only has the IndexWriter's commit() method called and the IndexReader is reopened. in order to keep lucene performance, we should not do commit more frequently, the best choice is more than 500 Documents has added to the index directory we call the commit method. 

 

PS : for some reason, the most is my english is not so proficient, so some of my viewpoint is not state clearly, Welcome your comments to made the view more clearly.