ElasticSearch(7)filebeat

ElasticSearch(7)filebeat

How to sync the date and time on ubuntu 18.04
https://linuxconfig.org/how-to-sync-time-on-ubuntu-18-04-bionic-beaver-linux

> sudo timedatectl set-ntp off
> timedatectl
                      Local time: Fri 2019-05-24 10:38:53 CDT
                  Universal time: Fri 2019-05-24 15:38:53 UTC
                        RTC time: Fri 2019-05-24 15:38:55
                       Time zone: America/Chicago (CDT, -0500)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no
> sudo timedatectl set-ntp on

Get the filebeat software first
https://www.elastic.co/downloads/beats/filebeat

> wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.1.0-linux-x86_64.tar.gz

Unzip the file and place it in working directory
> mv filebeat-7.1.0-linux-x86_64 ~/tool/filebeat-7.1.0

> sudo ln -s /home/carl/tool/filebeat-7.1.0 /opt/filebeat-7.1.0

> sudo ln -s /opt/filebeat-7.1.0 /opt/filebeat


There are AuditBeat, MetricBeat, FileBeat, PacketBeat, HeartBeat, WinLogBeat.

Create a Filebeat configuration file as follow:
> cat filebeat.yml
filebeat.inputs:
- input_type: log
  paths: /var/log/system1.log
  fields:
    name: systemLog
    level: debug
- input_type: log
  paths: /var/log/web1.log
  fields:
    name: webLog
    level: debug
output.logstash:
  hosts: ["ubuntu-master:5044"]

The purpose of this configuration is to use fields to put logging into different category and log level. Send all events to Logstash 5044 ports.

Fix the file permission of the configuration file
> chmod go-w /opt/filebeat/filebeat.yml

Start to run the Filebeat
> ./filebeat -e -c filebeat.yml

Once filebeat is running, we can set up the Logstash according to that as follow:
> cat config/logstash-filebeat.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
  beats {
    port => 5044
  }
}
output {
  stdout {}
  if ([fields][name] == "systemLog") {
    elasticsearch {
    hosts => ["http://ubuntu-master:9200"]
    index => "demolog1-%{+YYYY.MM.dd}"
    }
  }
  if ([fields][name] == "webLog") {
    elasticsearch {
      hosts => ["http://ubuntu-master:9200"]
      index => "demolog2-%{+YYYY-MM-dd}"
    }
  }
}

stdout is optional, I just want the standard output to show me the logging.
We can use fields name and level to decide which index in ElasticSearch Cluster it can go.


References:
https://www.jianshu.com/p/1133dd96f9bd
https://linuxconfig.org/how-to-sync-time-on-ubuntu-18-04-bionic-beaver-linux
https://www.jianshu.com/p/39453a671ce8
https://cloud.tencent.com/developer/article/1006051
https://www.ibm.com/developerworks/cn/opensource/os-cn-elk-filebeat/index.html
https://www.cnblogs.com/cjsblog/p/9445792.html
https://www.elastic.co/guide/en/beats/filebeat/7.1/logstash-output.html
https://discuss.elastic.co/t/filter-multiple-different-file-beat-logs-in-logstash/76847/4