sphinx coreseek 使用安装的简单描述
1、简介
1.1.Sphinx是什么
Sphinx是由俄罗斯人Andrew Aksyonoff开发的一个全文检索引擎。意图为其他应用提供高速、低空间占用、高结果 相关度的全文搜索功能。Sphinx可以非常容易的与SQL数据库和脚本语言集成。当前系统内置MySQL和PostgreSQL 数据库数据源的支持,也支持从标准输入读取特定格式 的XML数据。通过修改源代码,用户可以自行增加新的数据源(例如:其他类型的DBMS 的原生支持)
1.2.Sphinx的特性
(1)、高速的建立索引(在当代CPU上,峰值性能可达到10 MB/秒);
(2)、高性能的搜索(在2 – 4GB 的文本数据上,平均每次检索响应时间小于0.1秒);
(3)、可处理海量数据(目前已知可以处理超过100 GB的文本数据, 在单一CPU的系统上可 处理100 M 文档);
(4)、提供了优秀的相关度算法,基于短语相似度和统计(BM25)的复合Ranking方法;
(5)、支持分布式搜索;
(6)、支持短语搜索
(7)、提供文档摘要生成
(8)、可作为MySQL的存储引擎提供搜索服务;
(9)、支持布尔、短语、词语相似度等多种检索模式;
(10)、文档支持多个全文检索字段(最大不超过32个);
(11)、文档支持多个额外的属性信息(例如:分组信息,时间戳等);
(12)、支持断词;
1.3.Sphinx中文分词
中文的全文检索和英文等latin系列不一样,后者是根据空格等特殊字符来断词,而中文是根据语义来分词。目前大多数数据库尚未支持中文全文检索,如Mysql。故,国内出现了一些Mysql的中文全文检索的插件,做的比较好的有hightman的中文分词。Sphinx如果需要对中文进行全文检索,也得需要一些插件来补充。其中我知道的插件有 coreseek 和 sfc 。
2、安装配置实例
2.1 在GNU/Linux/unix系统上安装
2.1.1 sphinx安装
下载包到/usr/local/src目录下。
tar zxvf sphinx-2.0.8-release.tar.gz
cd sphinx-2.0.8-release
./configure –prefix=/usr/local/sphinx #注意:这里sphinx已经默认支持了mysql
make && make install # 其中的“警告”可以忽略
tar xzvf coreseek-3.2.14.tar.gz
cd coreseek-3.2.14 安装mmseg中文分词 cd mmseg-3.2.14 ./bootstrap #输出的warning信息可以忽视,若是呈现error则须要解决 ./configure --prefix=/usr/local/mmseg3 make && make install cd .. ##安装coreseek cd csft-3.2.14 sh buildconf.sh #输出的warning信息可以忽视,若是呈现error则须要解决 ./configure --prefix=/usr/local/coreseek --without-unixodbc --with-mmseg --with-mmseg-i ncludes=/usr/local/mmseg3/include/mmseg/ --with-mmseg-libs=/usr/local/mmseg3/lib/ --with-mysql ##若是提示mysql题目,可以查看MySQL数据源安装申明 make && make install
cd ..
生成和使用分词字典
bin/mmseg -u /usr/local/mmseg3/etc/unigram.txt
将生成unigram.txt.lib 文件
cp unigram.txt.lib /usr/local/mmseg3/etc/uni.lib
cd ../coreseek
[root@localhost coreseek]# mkdir dict ###创建字典目录
cp /usr/local/mmseg3/etc/unigram.txt.uni dict/uni.lib ###把创建的词典复制到dict
sudo vim dict/mmseg.ini ####创建mmseg的配置文件
mmseg.ini:
[mmseg]
merge_number_and_ascii=1;
number_and_ascii_joint=-;
compress_space=0;
seperate_number_ascii=1;
至此,mmseg配置完毕!下一步配置csft.conf——coreseek的配置文件
2.1.2 配置文件
cd /usr/local/sphinx/etc #进入sphinx的配置文件目录
cp sphinx.conf.dist sphinx.conf #新建Sphinx配置文件
sudo gedit sphinx.conf #编辑sphinx.conf
2.1.3 配置内容
(1)数据库的配置
(2)异常 #exceptions= /data/exceptions.txt 注释掉
(3) sql_query_pre= SET NAMES utf8去掉注释
(4)加一句: charset_dictpath = /usr/local/mmseg3/etc
(5) 变成:charset_type= zh_cn.utf-8
2.1.4 使用命令
#建立索引
/usr/local/coreseek/bin/indexer --all
/usr/local/coreseek/bin/search -c /usr/local/csft/etc/csft.conf 贾
3、 配置实例
设备MYSQL数据源
vi /usr/local/coreseek/etc/csft.conf
摘录我的MYSQL数据源设备文件
source src1 { # data source type. mandatory, no default value # known types are mysql, pgsql, mssql, xmlpipe, xmlpipe2, odbc type = mysql ##################################################################### ## SQL settings (for 'mysql' and 'pgsql' types) ##################################################################### # some straightforward parameters for SQL source types sql_host = localhost sql_user = root sql_pass = sql_db = test sql_port = 3306 # optional, default is 3306 # UNIX socket name # optional, default is empty (reuse client library defaults) # usually '/var/lib/mysql/mysql.sock' on Linux # usually '/tmp/mysql.sock' on FreeBSD # # sql_sock = /tmp/mysql.sock # MySQL specific client connection flags # optional, default is 0 # # mysql_connect_flags = 32 # enable compression # MySQL specific SSL certificate settings # optional, defaults are empty # # mysql_ssl_cert = /etc/ssl/client-cert.pem # mysql_ssl_key = /etc/ssl/client-key.pem # mysql_ssl_ca = /etc/ssl/cacert.pem # MS SQL specific Windows authentication mode flag # MUST be in sync with charset_type index-level setting # optional, default is 0 # # mssql_winauth = 1 # use currently logged on user credentials # MS SQL specific Unicode indexing flag # optional, default is 0 (request SBCS data) # # mssql_unicode = 1 # request Unicode data from server # ODBC specific DSN (data source name) # mandatory for odbc source type, no default value # # odbc_dsn = DBQ=C:\data;DefaultDir=C:\data;Driver={Microsoft Text Driver (*.txt; *.csv)}; # sql_query = SELECT id, data FROM documents.csv # pre-query, executed before the main fetch query # multi-value, optional, default is empty list of queries # sql_query_pre = SET NAMES utf8 # sql_query_pre = SET SESSION query_cache_type=OFF # main document fetch query # mandatory, integer document ID field MUST be the first selected column sql_query = \ SELECT goods_id,goods_name,goods_color\ FROM goods_test # range query setup, query that must return min and max ID values # optional, default is empty # # sql_query will need to reference $start and $end boundaries # if using ranged query: # # sql_query = \ # SELECT doc.id, doc.id AS group, doc.title, doc.data \ # FROM documents doc \ # WHERE id>=$start AND id<=$end # # sql_query_range = SELECT MIN(id),MAX(id) FROM documents # range query step # optional, default is 1024 # # sql_range_step = 1000 # unsigned integer attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # optional bit size can be specified, default is 32 # # sql_attr_uint = author_id # sql_attr_uint = forum_id:9 # 9 bits for forum_id sql_attr_uint = group_id # boolean attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # equivalent to sql_attr_uint with 1-bit size # # sql_attr_bool = is_deleted # bigint attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # declares a signed (unlike uint!) 64-bit attribute # # sql_attr_bigint = my_bigint_id # UNIX timestamp attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # similar to integer, but can also be used in date functions # # sql_attr_timestamp = posted_ts # sql_attr_timestamp = last_edited_ts sql_attr_timestamp = date_added # string ordinal attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # sorts strings (bytewise), and stores their indexes in the sorted list # sorting by this attr is equivalent to sorting by the original strings # # sql_attr_str2ordinal = author_name # floating point attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # values are stored in single precision, 32-bit IEEE 754 format # # sql_attr_float = lat_radians # sql_attr_float = long_radians # multi-valued attribute (MVA) attribute declaration # multi-value (an arbitrary number of attributes is allowed), optional # MVA values are variable length lists of unsigned 32-bit integers # # syntax is ATTR-TYPE ATTR-NAME 'from' SOURCE-TYPE [;QUERY] [;RANGE-QUERY] # ATTR-TYPE is 'uint' or 'timestamp' # SOURCE-TYPE is 'field', 'query', or 'ranged-query' # QUERY is SQL query used to fetch all ( docid, attrvalue ) pairs # RANGE-QUERY is SQL query used to fetch min and max ID values, similar to 'sql_query_range' # # sql_attr_multi = uint tag from query; SELECT id, tag FROM tags # sql_attr_multi = uint tag from ranged-query; \ # SELECT id, tag FROM tags WHERE id>=$start AND id<=$end; \ # SELECT MIN(id), MAX(id) FROM tags # post-query, executed on sql_query completion # optional, default is empty # # sql_query_post = # post-index-query, executed on successful indexing completion # optional, default is empty # $maxid expands to max document ID actually fetched from DB # # sql_query_post_index = REPLACE INTO counters ( id, val ) \ # VALUES ( 'max_indexed_id', $maxid ) # ranged query throttling, in milliseconds # optional, default is 0 which means no delay # enforces given delay before each query step sql_ranged_throttle = 0 # document info query, ONLY for CLI search (ie. testing and debugging) # optional, default is empty # must contain $id macro and must fetch the document by that id sql_query_info = SELECT * FROM documents WHERE id=$id # kill-list query, fetches the document IDs for kill-list # k-list will suppress matches from preceding indexes in the same query # optional, default is empty # # sql_query_killlist = SELECT id FROM documents WHERE edited>=@last_reindex # columns to unpack on indexer side when indexing # multi-value, optional, default is empty list # # unpack_zlib = zlib_column # unpack_mysqlcompress = compressed_column # unpack_mysqlcompress = compressed_column_2 # maximum unpacked length allowed in MySQL COMPRESS() unpacker # optional, default is 16M # # unpack_mysqlcompress_maxsize = 16M ##################################################################### ## xmlpipe settings ##################################################################### # type = xmlpipe # shell command to invoke xmlpipe stream producer # mandatory # # xmlpipe_command = cat /usr/local/coreseek/var/test.xml ##################################################################### ## xmlpipe2 settings ##################################################################### # type = xmlpipe2 # xmlpipe_command = cat /usr/local/coreseek/var/test2.xml # xmlpipe2 field declaration # multi-value, optional, default is empty # # xmlpipe_field = subject # xmlpipe_field = content # xmlpipe2 attribute declaration # multi-value, optional, default is empty # all xmlpipe_attr_XXX options are fully similar to sql_attr_XXX # # xmlpipe_attr_timestamp = published # xmlpipe_attr_uint = author_id # perform UTF-8 validation, and filter out incorrect codes # avoids XML parser choking on non-UTF-8 documents # optional, default is 0 # # xmlpipe_fixup_utf8 = 1 } # inherited source example # # all the parameters are copied from the parent source, # and may then be overridden in this source definition source src1throttled : src1 { sql_ranged_throttle = 100 } ############################################################################# ## index definition ############################################################################# # local index example # # this is an index which is stored locally in the filesystem # # all indexing-time options (such as morphology and charsets) # are configured per local index index test1 { # document source(s) to index # multi-value, mandatory # document IDs must be globally unique across all sources source = src1 # index files path and file name, without extension # mandatory, path must be writable, extensions will be auto-appended path = /usr/local/coreseek/var/data/test1 # document attribute values (docinfo) storage mode # optional, default is 'extern' # known values are 'none', 'extern' and 'inline' docinfo = extern # memory locking for cached data (.spa and .spi), to prevent swapping # optional, default is 0 (do not mlock) # requires searchd to be run from root mlock = 0 # a list of morphology preprocessors to apply # optional, default is empty # # builtin preprocessors are 'none', 'stem_en', 'stem_ru', 'stem_enru', # 'soundex', and 'metaphone'; additional preprocessors available from # libstemmer are 'libstemmer_XXX', where XXX is algorithm code # (see libstemmer_c/libstemmer/modules.txt) # # morphology = stem_en, stem_ru, soundex # morphology = libstemmer_german # morphology = libstemmer_sv morphology = none # minimum word length at which to enable stemming # optional, default is 1 (stem everything) # # min_stemming_len = 1 # stopword files list (space separated) # optional, default is empty # contents are plain text, charset_table and stemming are both applied # #stopwords = G:\data\stopwords.txt # wordforms file, in "mapfrom > mapto" plain text format # optional, default is empty # #wordforms = G:\data\wordforms.txt # tokenizing exceptions file # optional, default is empty # # plain text, case sensitive, space insensitive in map-from part # one "Map Several Words => ToASingleOne" entry per line # #exceptions = /data/exceptions.txt # minimum indexed word length # default is 1 (index everything) min_word_len = 1 charset_dictpath = /usr/local/coreseek/dict/ # charset encoding type # optional, default is 'sbcs' # known types are 'sbcs' (Single Byte CharSet) and 'utf-8' charset_type = zh_cn.utf-8 # charset definition and case folding rules "table" # optional, default value depends on charset_type # # defaults are configured to include English and Russian characters only # you need to change the table to include additional ones # this behavior MAY change in future versions # # 'sbcs' default value is # charset_table = 0..9, A..Z->a..z, _, a..z, U+A8->U+B8, U+B8, U+C0..U+DF->U+E0..U+FF, U+E0..U+FF # # 'utf-8' default value is # charset_table = 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F # ignored characters list # optional, default value is empty # # ignore_chars = U+00AD # minimum word prefix length to index # optional, default is 0 (do not index prefixes) # # min_prefix_len = 0 # minimum word infix length to index # optional, default is 0 (do not index infixes) # # min_infix_len = 0 # list of fields to limit prefix/infix indexing to # optional, default value is empty (index all fields in prefix/infix mode) # # prefix_fields = filename # infix_fields = url, domain # enable star-syntax (wildcards) when searching prefix/infix indexes # known values are 0 and 1 # optional, default is 0 (do not use wildcard syntax) # # enable_star = 1 # n-gram length to index, for CJK indexing # only supports 0 and 1 for now, other lengths to be implemented # optional, default is 0 (disable n-grams) # # ngram_len = 1 # n-gram characters list, for CJK indexing # optional, default is empty # # ngram_chars = U+3000..U+2FA1F # phrase boundary characters list # optional, default is empty # # phrase_boundary = ., ?, !, U+2026 # horizontal ellipsis # phrase boundary word position increment # optional, default is 0 # # phrase_boundary_step = 100 # whether to strip HTML tags from incoming documents # known values are 0 (do not strip) and 1 (do strip) # optional, default is 0 html_strip = 0 # what HTML attributes to index if stripping HTML # optional, default is empty (do not index anything) # # html_index_attrs = img=alt,title; a=title; # what HTML elements contents to strip # optional, default is empty (do not strip element contents) # # html_remove_elements = style, script # whether to preopen index data files on startup # optional, default is 0 (do not preopen), searchd-only # # preopen = 1 # whether to keep dictionary (.spi) on disk, or cache it in RAM # optional, default is 0 (cache in RAM), searchd-only # # ondisk_dict = 1 # whether to enable in-place inversion (2x less disk, 90-95% speed) # optional, default is 0 (use separate temporary files), indexer-only # # inplace_enable = 1 # in-place fine-tuning options # optional, defaults are listed below # # inplace_hit_gap = 0 # preallocated hitlist gap size # inplace_docinfo_gap = 0 # preallocated docinfo gap size # inplace_reloc_factor = 0.1 # relocation buffer size within arena # inplace_write_factor = 0.1 # write buffer size within arena # whether to index original keywords along with stemmed versions # enables "=exactform" operator to work # optional, default is 0 # # index_exact_words = 1 # position increment on overshort (less that min_word_len) words # optional, allowed values are 0 and 1, default is 1 # # overshort_step = 1 # position increment on stopword # optional, allowed values are 0 and 1, default is 1 # # stopword_step = 1 } # inherited index example # # all the parameters are copied from the parent index, # and may then be overridden in this index definition index test1stemmed : test1 { path = /usr/local/coreseek/var/data/test1stemmed morphology = stem_en } # distributed index example # # this is a virtual index which can NOT be directly indexed, # and only contains references to other local and/or remote indexes index dist1 { # 'distributed' index type MUST be specified type = distributed # local index to be searched # there can be many local indexes configured local = test1 local = test1stemmed # remote agent # multiple remote agents may be specified # syntax for TCP connections is 'hostname:port:index1,[index2[,...]]' # syntax for local UNIX connections is '/path/to/socket:index1,[index2[,...]]' agent = localhost:9313:remote1 agent = localhost:9314:remote2,remote3 # agent = /var/run/searchd.sock:remote4 # blackhole remote agent, for debugging/testing # network errors and search results will be ignored # # agent_blackhole = testbox:9312:testindex1,testindex2 # remote agent connection timeout, milliseconds # optional, default is 1000 ms, ie. 1 sec agent_connect_timeout = 1000 # remote agent query timeout, milliseconds # optional, default is 3000 ms, ie. 3 sec agent_query_timeout = 3000 } ############################################################################# ## indexer settings ############################################################################# indexer { # memory limit, in bytes, kiloytes (16384K) or megabytes (256M) # optional, default is 32M, max is 2047M, recommended is 256M to 1024M mem_limit = 32M # maximum IO calls per second (for I/O throttling) # optional, default is 0 (unlimited) # # max_iops = 40 # maximum IO call size, bytes (for I/O throttling) # optional, default is 0 (unlimited) # # max_iosize = 1048576 # maximum xmlpipe2 field length, bytes # optional, default is 2M # # max_xmlpipe2_field = 4M # write buffer size, bytes # several (currently up to 4) buffers will be allocated # write buffers are allocated in addition to mem_limit # optional, default is 1M # # write_buffer = 1M } ############################################################################# ## searchd settings ############################################################################# searchd { # hostname, port, or hostname:port, or /unix/socket/path to listen on # multi-value, multiple listen points are allowed # optional, default is 0.0.0.0:9312 (listen on all interfaces, port 9312) # # listen = 127.0.0.1 # listen = 192.168.0.1:9312 # listen = 9312 # listen = /var/run/searchd.sock # log file, searchd run info is logged here # optional, default is 'searchd.log' log = /usr/local/coreseek/var/log/searchd.log # query log file, all search queries are logged here # optional, default is empty (do not log queries) query_log = /usr/local/coreseek/var/log/query.log # client read timeout, seconds # optional, default is 5 read_timeout = 5 # request timeout, seconds # optional, default is 5 minutes client_timeout = 300 # maximum amount of children to fork (concurrent searches to run) # optional, default is 0 (unlimited) max_children = 30 # PID file, searchd process ID file name # mandatory pid_file = /usr/local/coreseek/var/log/searchd.pid # max amount of matches the daemon ever keeps in RAM, per-index # WARNING, THERE'S ALSO PER-QUERY LIMIT, SEE SetLimits() API CALL # default is 1000 (just like Google) max_matches = 1000 # seamless rotate, prevents rotate stalls if precaching huge datasets # optional, default is 1 seamless_rotate = 1 # whether to forcibly preopen all indexes on startup # optional, default is 0 (do not preopen) preopen_indexes = 0 # whether to unlink .old index copies on succesful rotation. # optional, default is 1 (do unlink) unlink_old = 1 # attribute updates periodic flush timeout, seconds # updates will be automatically dumped to disk this frequently # optional, default is 0 (disable periodic flush) # # attr_flush_period = 900 # instance-wide ondisk_dict defaults (per-index value take precedence) # optional, default is 0 (precache all dictionaries in RAM) # # ondisk_dict_default = 1 # MVA updates pool size # shared between all instances of searchd, disables attr flushes! # optional, default size is 1M mva_updates_pool = 1M # max allowed network packet size # limits both query packets from clients, and responses from agents # optional, default size is 8M max_packet_size = 8M # crash log path # searchd will (try to) log crashed query to 'crash_log_path.PID' file # optional, default is empty (do not create crash logs) # # crash_log_path = /usr/local/coreseek/var/log/crash # max allowed per-query filter count # optional, default is 256 max_filters = 256 # max allowed per-filter values count # optional, default is 4096 max_filter_values = 4096 # socket listen queue length # optional, default is 5 # # listen_backlog = 5 # per-keyword read buffer size # optional, default is 256K # # read_buffer = 256K # unhinted read size (currently used when reading hits) # optional, default is 32K # # read_unhinted = 32K }
a. source是设备数据源,遵守提示输入MYSQL的主机、帐号、暗码和数据库即可,我的MYSQL就安装在本机上(MYSQL的安装可自行百度)
b. sql_query_pre是在履行查询之前履行的SQL语句。(重视:在coreseek只能辨认utf8字符集编码,所以我们要履行转换一下)
c. sql_query是要查询进行索引的SQL语句,sql_attr_unit和sql_attr_timestamp是设置属性的,属性在全文检索中可以用来设置过滤和排序。
d. index和source应当是成对呈现,index就是设备索引的功能(我们还可以设备多个索引 主索引+增量索引的功能)
e. searchd是常驻过程的全文检索办事,默认监控本机的9312端口
f. charset_type和charset_dictpath是中文分词设备
4、应用
4.1 在CLI上测试
建立索引: /usr/local/coreseek/bin/indexer --all
搜索中文:/usr/local/coreseek/bin/search -c /usr/local/coreseek/etc/csft.conf '贾'
5、附录
相关推荐
在描述中提到了一个博文链接,虽然具体内容没有给出,但可以推测博主“sinykk”分享了他在安装和配置Sphinx以及与Coreseek集成过程中的经验。由于没有实际的博文内容,我将基于一般流程来解释Sphinx的安装和使用。 ...
《Sphinx 0.9.9/Coreseek 3.2 参考手册》是一本详细介绍Sphinx和Coreseek这两款全文检索引擎的指南。Sphinx是一款功能强大的开源全文搜索引擎,而Coreseek则是在Sphinx的基础上,针对中文环境进行了优化的免费开源...
博文链接提供的资源可能包含作者对CoreSeek的详细使用经验、安装配置教程、问题解决策略以及性能优化技巧。由于描述部分是NULL,我们无法直接获取具体信息,但通常这类博文会涵盖以下几个方面的知识点: 1. **...
3. **索引构建**:在“sphinx索引”这一描述中,提到的是Sphinx的索引构建过程。在使用CoreSeek时,首先需要对数据进行索引,将原始数据转化为可快速查询的结构。这个过程包括数据抽取、预处理(如分词)、倒排索引...
1. 安装:正确安装Sphinx和相关依赖,如CoreSeek。 2. 配置:修改`sphinx.conf`,设置数据源、索引、搜索日志等参数,特别是数据库连接信息,以适应你的需求。 3. 创建索引:使用`sphinxbuild`工具或者配置文件中的...
虽然描述中提到不再详述具体使用方法,但一般来说,安装和配置CoreSeek涉及以下几个步骤: 1. 解压并安装CoreSeek服务器。 2. 配置CoreSeek与MySQL的连接。 3. 安装和配置分词插件。 4. 创建索引,将需要搜索的数据...
README.txt文件通常包含了安装和使用指南,是用户开始使用Coreseek的重要参考文档。"新建文本文档.txt"可能是开发者留下的临时文件或未命名的说明,具体用途可能需要查看其内容。"bin"目录通常包含可执行程序,"var...
2. **Coreseek全文检索引擎**:Coreseek是一个开源的全文检索解决方案,它基于Sphinx搜索引擎,具有高速度和易于扩展的特点。它通常用于构建高性能的全文检索系统。 3. **Python**:Python是一种广泛使用的高级编程...
国内的Coreseek项目基于Sphinx开发,增加了对中文分词和编码的支持,并优化了中文搜索结果的排序。 2. MySQL数据库与分词: MySQL是一个广泛使用的开源关系数据库管理系统,支持英文分词的全文检索,但对于中文分词...
此外,相关的扩展阅读文章提到了Coreseek(Sphinx的中文版)、Sphinx的安装、配置以及使用示例,还涵盖了如何将Sphinx与MySQL结合进行多线程搜索、生成PDF文档,以及在PHP中启用Sphinx全文搜索的方法,这些都是...
在描述中提到的"支持coreseek实时索引"这一点,CoreSeek是一个基于Sphinx搜索库的开源全文搜索引擎,适用于PHP环境。它提供了强大的全文检索、布尔运算、短语搜索、模糊匹配等功能,并且支持实时索引,即在数据插入...