ElasticSearch配置文档(v1.7.2)

chengyuanheng

浏览: 19581 次
性别:
来自: 北京

最近访客更多访客>>

lxzqz

szs110

amo

karbo

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

搜索引擎

搜索引擎

ElasticSearch

安装java环境

sudo apt-get update
java -version
sudo apt-get install default-jre
sudo apt-get install default-jdk
（或者： yum -y install java-1.7.0-openjdk* ）

设置JAVA_HOME

sudo update-alternatives --config java（返回YOUR_PATH）
sudo nano /etc/environment
JAVA_HOME="YOUR_PATH"
source /etc/environment
echo $JAVA_HOME

安装ElasticSearch（以1.7.2为例）

wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb(yum 下载rpm包)
sudo dpkg -i elasticsearch-1.7.2.deb (sudo rpm -ivh *.rpm)
sudo update-rc.d elasticsearch defaults (chkconfig –-add elasticsearch ; chkconfig elasticsearch on )

启动ElasticSearch

sudo service elasticsearch start
elasticsearch 结构目录位置： /var/lib/elasticsearch

配置ElasticSearch

配置文件地址：/etc/elasticsearch/elasticsearch.yml
配置文件详解：http://www.cnblogs.com/sunxucool/p/3799190.html

配置防火墙

sudo ufw disable
sudo ufw allow from 192.168.1.141
sudo ufw enable

或者防火墙

#/usr/sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEPT

#/usr/sbin/iptables -I INPUT -p tcp --dport 22 -j ACCEPT

#/usr/sbin/iptables -I INPUT -p tcp --dport 3306 -j ACCEPT

#/usr/sbin/iptables -I INPUT -p tcp --dport 8080 -j ACCEPT

/usr/sbin/iptables -I INPUT -p tcp --dport 9200 -j DROP

/usr/sbin/iptables -I INPUT -s 10.44.136.154 -p tcp --dport 9200 -j ACCEPT

/usr/sbin/iptables -I INPUT -s 某个ip -p tcp --dport 9200 -j ACCEPT

/usr/sbin/service iptables save

创建快照

elasticsearch.yml添加： path.repo: ["/mount/backups", "/mount/longterm_backups"]
创建目录： /mount/backups/my_backup, /mount/longterm_backups
添加权限： sudo chmod -R 777 /mount
重启elasticsearch服务： sudo service elasticsearch restart
创建仓库，并指定仓库类型：

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '

{

"type": "fs",

"settings": {

"location": "/mount/backups/my_backup",

"compress": true

}

}'

创建第一个快照：curl -XPUT http://localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true

查看所有快照： curl -XGET http://localhost:9200/_snapshot/my_backup/_all
删除一个快照： curl -XDELETE http://localhost:9200/_snapshot/my_backup/snapshot_1
使用快照：

curl -XPOST http://localhost:9200/_snapshot/my_backup/snapshot_1/_restore -d '

{

"indices": "index_1,index_2",

"ignore_unavailable": "true",

"include_global_state": false,

"rename_pattern": "index_(.+)",

"rename_replacement": "restored_index_$1"

}'

配置ik分词

版本参考

IK version ES version

master	2.1.0 -> master
1.6.0	2.1.0
1.5.0	2.0.0
1.4.1	1.7.2
1.4.0	1.6.0
1.3.0	1.5.0
1.2.9	1.4.0
1.2.8	1.3.2
1.2.7	1.2.1
1.2.6	1.0.0
1.2.5	0.90.2
1.2.3	0.90.2
1.2.0	0.90.0
1.1.3	0.20.2
1.1.2	0.19.x
1.0.0	0.16.2 -> 0.19.0

克隆ik库： https://github.com/medcl/elasticsearch-analysis-ik.git

checkout对应tag： git checkout tags/v1.4.1 -b v1.4.1

安装maven： (略)

编译jar包： sudo mvn package

copy ik文件夹到/etc/elasticsearch/: sudo cp config/ik /etc/elasticsearch/

copy jar到/usr/share/elasticsearch/lib/: sudo cp target/*.jar /usr/share/elasticsearch/lib

配置elasticsearch.yml

index:
  analysis:
    analyzer:
      ik:
        alias: [ik_analyzer]
        type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
        type: ik
        use_smart: false
      ik_smart:
        type: ik
        use_smart: true

index.analysis.analyzer.default.type : "ik"

创建index:

curl -XPUT http://localhost:9200/index

创建mapping

curl -XPOST http://localhost:9200/index/fulltext/_mapping -d'
{
    "fulltext": {
             "_all": {
            "analyzer": "ik_max_word",
            "search_analyzer": "ik_max_word",
            "term_vector": "no",
            "store": "false"
        },
        "properties": {
            "content": {
                "type": "string",
                "store": "no",
                "term_vector": "with_positions_offsets",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_max_word",
                "include_in_all": "true",
                "boost": 8
            }
        }
    }
}'

mapping:

curl -XPUT localhost:9200/feiliwu -d '{

"mappings":{

"product":{

"_timestamp":{

"enables": true

}

录入数据

curl -XPOST http://localhost:9200/index/fulltext/1 -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'

curl -XPOST http://localhost:9200/index/fulltext/2 -d'
{"content":"公安部：各地校车将享最高路权"}
'

curl -XPOST http://localhost:9200/index/fulltext/3 -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'

curl -XPOST http://localhost:9200/index/fulltext/4 -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'

查询

curl -XPOST http://localhost:9200/index/fulltext/_search  -d'
{
    "query" : { "term" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "content" : {}
        }
    }
}
'

安装Marvel

run sudo ./bin/plugin -i elasticsearch/marvel/latest
浏览器访问：http://localhost:9200/_plugin/marvel/

参考资料

Logstash

下载安装

wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.4-1_all.deb
sudo dpkg -i logstash_1.5.4-1_all.deb
sudo update-rc.d logstash defaults

配置jdk

sudo nano /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar

export PATH=$PATH:$JAVA_HOME/bin

export LOGSTASH_HOME=/opt/logstash

export LOGSTASH=$LOGSTASH_HOME/bin

export PATH=$LOGSTASH:$PATH

source /etc/profile

配置logstash

sudo nano /etc/logstash/conf.d/logstash-simple.conf

input {

tcp {

port => 5000

type => syslog

}

udp {

port => 5000

type => syslog

}

filter {

if [type] == "syslog" {

grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]

add_field => [ "received_from", "%{host}" ]

}

syslog_pri { }

date {

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

}

output {

elasticsearch {

protocol => "http"

host => "localhost:9200"

}

stdout { }

}

启动logstash

sudo service logstash status
sudo service logstash start

查看log

tail -f /var/log/logstash/logstash.log

参考资料

Kibana

建立使用kibana的分组

sudo groupadd -g 999 kibana
sudo useradd -u 999 -g 999 kibana
If those commands fail because the999GID or UID already exist, replace the number with IDs that are free

下载安装

cd ~
wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
tar xvf kibana-*.tar.gz

配置

sudo nano ~/kibana-4*/config/kibana.yml（server.host: "localhost"）
sudo mkdir -p /opt/kibana
sudo cp -R ~/kibana-4*/* /opt/kibana/
sudo chown -R kibana: /opt/kibana

run as a service

cd /etc/init.d && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init
cd /etc/default && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default
sudo chmod +x /etc/init.d/kibana
sudo update-rc.d kibana defaults 96 9
sudo service kibana start
visit localhost:5601

Install Nginx

sudo apt-get install nginx apache2-utils(sudo yum install nginx httpd)
sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin
sudo vi /etc/nginx/sites-available/default

server {

listen 80;

server_name localhost;

auth_basic "Restricted Access";

auth_basic_user_file /etc/nginx/htpasswd.users;

location / {

proxy_pass http://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

}