Apache Solr 实现去掉重复的搜索结果

　打上SOLR-236_collapsing.patch补丁，实现 solr 搜索结果折叠、除去重复的搜索结果，可以实现类似google搜索结果的“站内的其它相关信息 ”。solr collapsing patch 是用 hash 某个字段来实现折叠重复结果的。下面我演示下应用这个补丁并搜索试用下。

 

其实 solr 上已经有了这功能的实现：solr 1.3 collapse patch， 请看：https://issues.apache.org/jira/browse/SOLR-236 ，我这里下载是了新的：https://issues.apache.org/jira/secure/attachment/12403590/SOLR-236_collapsing.patch 。

下载 好后就需要打上补丁了，先准备一份源码在D:/apache-solr-1.3.0目录下。没有可以去下载：http: //archive.apache.org/dist/lucene/solr/1.3.0/apache-solr-1.3.0.zip。把SOLR- 236_collapsing.patch文件放在D:/apache-solr-1.3.0目录下， 打补丁有我知道的有两种：用linux工具 patch（windows 下有 cygwin）；用 ant 的 patch。

windows cygwin 的 patch：

D:\apache-solr-1.3.0>patch -p0 < SOLR-236_collapsing.patch
patching file src/test/org/apache/solr/search/TestDocSet.java
patching file src/java/org/apache/solr/search/CollapseFilter.java
patching file src/java/org/apache/solr/search/DocSet.java
patching file src/java/org/apache/solr/search/NegatedDocSet.java
patching file src/java/org/apache/solr/search/SolrIndexSearcher.java
patching file src/java/org/apache/solr/common/params/CollapseParams.java
patching file src/java/org/apache/solr/handler/component/CollapseComponent.java

ant patch，把下面的内容保存为 patch-build.xml 放到 D:\apache-solr-1.3.0 目录下：

<?
xml
 
version
=
“1.0″
 
encoding
=
“UTF-8″
?>
<
project
 
name
=
“solr-patch”
 
default
=
“apply-patch”
 
basedir
=
“.”
>

    <
target
 
name
=
“apply-patch”
 
description
=
“Apply a patch file.  Set -Dpatch.file”
>
        <
patch
 
patchfile
=
“${patch.file}”
 
strip
=
“0″
/>
    </
target
>
</
project
>

ant 打补丁：

D:\apache-solr-1.3.0>ant -Dpatch.file=SOLR-236_collapsing.patch -f patch-build.xml
Buildfile: patch-build.xml

apply-patch:
[patch] patching file src/test/org/apache/solr/search/TestDocSet.java
[patch] patching file src/java/org/apache/solr/search/CollapseFilter.java
[patch] patching file src/java/org/apache/solr/search/DocSet.java
[patch] patching file src/java/org/apache/solr/search/NegatedDocSet.java
[patch] patching file src/java/org/apache/solr/search/SolrIndexSearcher.java
[patch] patching file src/java/org/apache/solr/common/params/CollapseParams.java
[patch] patching file src/java/org/apache/solr/handler/component/CollapseComponent.java

BUILD SUCCESSFUL
Total time: 0 seconds

源码打上了补丁，然后用 ant 构建源码：

D:\apache-solr-1.3.0>ant dist

在 D:/apache-solr-1.3.0/dist 目录下可以找到编译好的 solr 了。然后把 solr 放到 tomcat 中去运行它，把下面的内容保存在 TOMCAT_HOME/conf/Catalina/localhost/solr.xml 文件中：

<
Context
 
docBase
=
“D:\apache-solr-1.3.0\dist\apache-solr-1.3.0.war”
 
reloadable
=
“true”
 
>
    <
Environment
 
name
=
“solr/home”
 
type
=
“java.lang.String”
 
value
=
“D:\apache-solr-1.3.0\example\solr”
 
override
=
“true”
 
/>
</
Context
>

修改 D:\apache-solr-1.3.0\example\solr\conf\solrconfig.xml 使 solr 可以支持 collapse。

定义搜索组件，在 QueryComponent 附近：

<
searchComponent
 
name
=
“collapse”
 
class
=
“org.apache.solr.handler.component.CollapseComponent”
 
/>

定义一个 handler 使用上面的搜索组件：

<
requestHandler
 
name
=
“collapse”
 
class
=
“solr.SearchHandler”
>
  <!– default values for query parameters –>
   <
lst
 
name
=
“defaults”
>
     <
str
 
name
=
“echoParams”
>
explicit
</
str
>
   </
lst
>
  <
arr
 
name
=
“components”
>
    <
str
>
collapse
</
str
>
    <
str
>
debug
</
str
>
  </
arr
>
</
requestHandler
>

安装启动 tomcat，现在提交一些数据给它，用官方的示例数据就可以了。运行：

D:\apache-solr-1.3.0\example\exampledocs>java -Durl=http://localhost:8080/solr/update -Dcommit=yes -jar post.jar *.xml
SimplePostTool: version 1.2
SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, other encodings are not currently supported
SimplePostTool: POSTing files to http://localhost:8080/solr/update..
SimplePostTool: POSTing file hd.xml
SimplePostTool: POSTing file ipod_other.xml
SimplePostTool: POSTing file ipod_video.xml
SimplePostTool: POSTing file mem.xml
SimplePostTool: POSTing file monitor.xml
SimplePostTool: POSTing file monitor2.xml
SimplePostTool: POSTing file mp500.xml
SimplePostTool: POSTing file sd500.xml
SimplePostTool: POSTing file solr.xml
SimplePostTool: POSTing file spellchecker.xml
SimplePostTool: POSTing file utf8-example.xml
SimplePostTool: POSTing file vidcard.xml
SimplePostTool: COMMITting Solr index changes..

http://localhost:8080/solr/admin/stats.jsp   有结果了？ 有了。然后开始查询试试看。

查询：http://localhost:8080/solr/select/?q=*%3A*&indent=on&qt=collapse&collapse=true&collapse.field=popularity&collapse.threshold=1

结果：

<?
xml
 
version
=
“1.0″
 
encoding
=
“UTF-8″
?>
<
response
>

<
lst
 
name
=
“responseHeader”
>
 <
int
 
name
=
“status”
>
0
</
int
>
 <
int
 
name
=
“QTime”
>
0
</
int
>
 <
lst
 
name
=
“params”
>
  <
str
 
name
=
“collapse.field”
>
popularity
</
str
>
  <
str
 
name
=
“fl”
>
id
</
str
>
  <
str
 
name
=
“collapse.threshold”
>
1
</
str
>
  <
str
 
name
=
“indent”
>
on
</
str
>
  <
str
 
name
=
“q”
>
*:*
</
str
>
  <
str
 
name
=
“qt”
>
collapse
</
str
>
  <
str
 
name
=
“collapse”
>
true
</
str
>
 </
lst
>
</
lst
>
<
lst
 
name
=
“collapse_counts”
>
 <
str
 
name
=
“field”
>
popularity
</
str
>
 <
lst
 
name
=
“doc”
>
  <
int
 
name
=
“SP2514N”
>
4
</
int
>
  <
int
 
name
=
“F8V7067-APL-KIT”
>
1
</
int
>
  <
int
 
name
=
“MA147LL/A”
>
1
</
int
>
  <
int
 
name
=
“TWINX2048-3200PRO”
>
1
</
int
>
  <
int
 
name
=
“VS1GB400C3″
>
3
</
int
>
  <
int
 
name
=
“1″
>
10
</
int
>
 </
lst
>
 <
lst
 
name
=
“count”
>
  <
int
 
name
=
“6″
>
4
</
int
>
  <
int
 
name
=
“1″
>
1
</
int
>
  <
int
 
name
=
“10″
>
1
</
int
>
  <
int
 
name
=
“5″
>
1
</
int
>
  <
int
 
name
=
“7″
>
3
</
int
>
  <
int
 
name
=
“0″
>
10
</
int
>
 </
lst
>
 <
str
 
name
=
“debug”
>
HashDocSet(6) Time(ms): 0/0/0/0
</
str
>
</
lst
>
<
result
 
name
=
“response”
 
numFound
=
“6″
 
start
=
“0″
>
 <
doc
>
  <
str
 
name
=
“id”
>
SP2514N
</
str
>
 </
doc
>
 <
doc
>
  <
str
 
name
=
“id”
>
F8V7067-APL-KIT
</
str
>
 </
doc
>
 <
doc
>
  <
str
 
name
=
“id”
>
MA147LL/A
</
str
>
 </
doc
>
 <
doc
>
  <
str
 
name
=
“id”
>
TWINX2048-3200PRO
</
str
>
 </
doc
>
 <
doc
>
  <
str
 
name
=
“id”
>
VS1GB400C3
</
str
>
 </
doc
>
 <
doc
>
  <
str
 
name
=
“id”
>
1
</
str
>
 </
doc
>
</
result
>
</
response
>

可以看到 collapse_counts 相关的输出：

<
lst
 
name
=
“collapse_counts”
>
 <
str
 
name
=
“field”
>
popularity
</
str
>
 <
lst
 
name
=
“doc”
>
  <
int
 
name
=
“SP2514N”
>
4
</
int
>
…
 </
lst
>
 <
lst
 
name
=
“count”
>
  <
int
 
name
=
“6″
>
4
</
int
>
  <
int
 
name
=
“1″
>
1
</
int
>
  <
int
 
name
=
“10″
>
1
</
int
>
  <
int
 
name
=
“5″
>
1
</
int
>
  <
int
 
name
=
“7″
>
3
</
int
>
  <
int
 
name
=
“0″
>
10
</
int
>
 </
lst
>
 <
str
 
name
=
“debug”
>
HashDocSet(6) Time(ms): 0/0/0/0
</
str
>
</
lst
>

上面的 count 下的内容（它的顺序是result/doc的顺序），表示 popularity=6 相同的结果还有 4 个，与 popularity=1 相同的结果还有 1 个，依此类推。这样就可以显示给用户的界面里提示“相同的其它内容不有N个”。

使用的参数有：

#启用 collapse 组件
collapse=true
#用那个字段来 hash 去除重量内容
collapse.field=popularity
#要结果中可以最多出现几个相同的文档
collapse.threshold=1

当然还有其它参数，请看：org.apache.solr.common.params.CollapseParams 类。

原文出处：http://blog.chenlb.com/2009/04/apply-solr-collapsing-patch-remove-duplicate-result.html