filebeat+elasticsearch+kibana(版本6.4.2)搭建简单的日志收集系统

1 elasticsearch

1.1.安装elasticsearch

使用官网rpm包

# rpm -ivh elasticsearch-6.4.2.rpm

1.2.修改elasticsearch 配置文件

# vi /etc/elasticsearch/elasticsearch.yml

 

关键配置：

cluster.name: XXX-application

node.name: node1

path.data:自定目录

path.logs: 自定义目录

bootstrap.memory_lock: true

network.host:ip或者0.0.0.0

http.port: 9200

# 设置节点间交互的tcp端口,默认是9300

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["ip:9300", "ip:9300"]

#两个节点防脑裂的配置, minimum_master_nodes决定了选主需要最少节点数, N/2+1, 两个节点即2

discovery.zen.minimum_master_nodes: 2

 

1.3.把elasticsearch目录改成elasticsearch用户elasticsearch组

chown -R elasticsearch:elasticsearch /usr/share/elasticsearch

 

path.data path.logs对应的目录进行同样的赋权操作

 

如果使用systemctl 启动不起可能也是与目录的用户归属有关

可以使用 find / -name elasticsearch 查看还有哪些elasticsearch目录

1.4.开发防火墙 端口 9200 9300

1.5修改elasticsearch 的jvm配置

vi /etc/sysconfig/elasticsearch

指定JAVA_HOME目录

JAVA_HOME=/usr/local/java/jdk1.8.0_171

修改jvm参数

vi /etc/elasticsearch/jvm.options

-Xms16g

-Xmx16g

 

## GC configuration

#-XX:+UseConcMarkSweepGC

#-XX:CMSInitiatingOccupancyFraction=75

#-XX:+UseCMSInitiatingOccupancyOnly

-XX:+UseG1GC

-XX:MaxGCPauseMillis=50

-XX:MaxGCPauseMillis=200

 

1.6.修改系统打开文件数据及内存锁相关配置

vi /etc/security/limits.conf

加上：

* soft nofile 65536

* hard nofile 65536

* soft nproc 32000

* hard nproc 32000

* hard memlock unlimited

* soft memlock unlimited

vi /etc/systemd/system.conf

加上：

DefaultLimitNOFILE=65536

DefaultLimitNPROC=32000

DefaultLimitMEMLOCK=infinity

之后继续操作:

/bin/systemctl daemon-reload

/bin/systemctl enable elasticsearch.service

systemctl start elasticsearch.service

systemctl status elasticsearch.service

 

尝试过一定要重启机器，不然还是会报

[1] bootstrap checks failed

[1]: memory locking requested for elasticsearch process but memory is not locked

 

1.7.安装geoip 插件

官网参考地址：https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/ingest-geoip.html

离线安装下载地址：

https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-geoip/ingest-geoip-6.4.2.zip

安装命令： bin/elasticsearch-plugin install file:///path/to/plugin.zip

# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///opt/elk/ingest-geoip-6.4.2.zip

修改 geoip插件目录归属elasticsearch用户 elasticsearch组

#chown elasticsearch:elasticsearch /usr/share/elasticsearch/plugins/ingest-geoip

 

2 filebeat

2.1 安装filebeat

#rpm -ivh filebeat-6.4.2-x86_64.rpm

2.2 修改 filebeat 配置：关键地方

vi /etc/filebeat/filebeat.yml

- type: log

enabled: true

paths:

- /opt/testLog/*.log

fields:

source: system

- type: log

enabled: false

paths:

- /opt/testLog1/*.log

fields:

source: system

 

output.elasticsearch:

# Array of hosts to connect to.

hosts: ["192.168.12.140:9200","192.168.12.141:9200"]

# Optional protocol and basic auth credentials.

#protocol: "https"

#username: "elastic"

#password: "changeme"

#pipeline: "nginx-pipeline"

#index: "mylog-%{+yyyy.MM.dd}"

pipelines:

- pipeline: "nginx-pipeline"

when.contains:

fields.source: nginx

- pipeline: "system-pipeline"

when.contains:

fields.source: system

indices:

- index: "mylog-nginx-%{+yyyy.MM.dd}"

when:

contains:

fields.source: nginx

- index: "mylog-system-%{+yyyy.MM.dd}"

when:

contains:

fields.source: system

setup.template.name: "mylog"

setup.template.pattern: "mylog-*"

 

备注：配文件里是source: nginx fsource: system,但是我在生产配置的时候，index跟pipeline一直不生效，怎么尝试都不行，后来了一下生产出来的数据，有另个对应的key是fields.source，改成fields.source: nginx fields.source: system就可以了。

 

3 kibana

安装kibana

rpm -ivh kibana-6.4.2-x86_64.rpm

kibana的启动配置好简单这么就不细说了

 

4. 在elasticsearch中建立 pipeline 与filebeat对应

4.1 建立 nginx.json文件

vi nginx.json

{ "description" : "nginx-pipeline",

"processors" : [

{

"grok" :{

"field" : "message",

"patterns" :["%{IPORHOST:clientip} - %{NOTSPACE:remote_user} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} \"(?:%{USERNAME:http_x_forwarded_for}|%{IPV4:http_x_forwarded_for})\""

],

"pattern_definitions" : {

"NGUSERNAME" : "[a-zA-Z.@-+_%]",

"NGUSER" : "%{NGUSERNAME}"

},

"ignore_failure" : true,

"ignore_missing" : true

 

}

},

{

"date": {

"field": "timestamp",

"target_field": "@timestamp",

"formats": [

"dd/MMM/yyyy:HH:mm:ss Z"

],

"timezone": "Asia/Shanghai"

}

}

]

}

注：这里的patterns要根据具体的日志文件来实现

 

4.2 把pipeline导入到elasticsearch

curl -H 'Content-Type: application/json' -XPUT 'http://192.168.12.140:9200/_ingest/pipeline/nginx-pipeline' -d@/etc/elasticsearch/nginx.json

这里的 nginx-pipeline 对应的是 filebeat里面的 - pipeline: "nginx-pipeline"

system-pipeline根据实际情况参考nginx.json进行配置

 

4.3关于grok Ingest 自带的正则模板可以参考这里

https://github.com/elastic/elasticsearch/tree/6.4/libs/grok/src/main/resources/patterns