ronin47

浏览: 983911 次
性别:
来自: 上海

最近访客更多访客>>

Sun_kiss

oszerone

hc_face

u012363178

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

elk

博客分类：

python&nodejs

监控

日志监控系统

Collectd: gather statistics on performance, processes, and overall status of the system
Graphite: Store numeric time-series data & Render graphs of this data on demand
Grafana: visualization and dashboarding tool for time series data

一个日志监控系统包括：日志收集，日志存储，日志可视化。
分别对应Collectd，Graphite（不用graphite本身的web工具），Grafana。
时间序列数据库的替代方案有：elasticSearch，influxdb。
而logstash可以认为是日志收集和日志存储之间的桥梁，因为它提供了I/O配置方便地在不同系统中进行数据传输。
如果没有logstash作为桥梁，日志收集后怎么放到存储中是个问题，需要自己调用客户端API。

那么这些系统之间如何通信，如何组织？

collectd负责收集数据，并通过network可以发送到logstash的指定端口
logstash的输入是监听步骤1的端口，输出可以写到存储系统中，比如es,influxdb
grafana通过配置数据源的方式，获取存储系统中的数据，进行可视化展现

Software Version 安装节点软件说明

elasticSearch	2.3.3	192.168.6.52	索引数据库
logstash	2.3.2	本机	有Input/Outout，因此可以连接各种管道
collectd		192.168.6.52	收集机器的信息，性能，进程等
Grafana		192.168.6.52	可视化，可以接入不同的数据源es和influxdb
influxdb	0.13	192.168.6.52	时间序列数据库

ELK

LogStash

标准I/O

命令行输入输出，通过脚本执行，codec指定如何解码

$ cd logstash-2.3.2
$ bin/logstash -e "input {stdin{}} output {stdout{}}" <<< 'Hello World'
Settings: Default pipeline workers: 4
Pipeline main started
2016-05-25T04:00:29.531Z zqhmac Hello World
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

$ bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}' <<< 'Hello World'
{
       "message" => "Hello World",
      "@version" => "1",
    "@timestamp" => "2016-05-25T04:01:09.095Z",
          "host" => "zqhmac"
}

$ vi logstash-simple.conf 
input { stdin {} }
output {
   stdout { codec=> rubydebug }
}
$ bin/logstash agent -f logstash-simple.conf --verbose

File Input

$ bin/logstash agent -f logstash-file.conf  --verbose
input {
    file {
        path => "/usr/install/cassandra/logs/system.log"
        start_position => beginning
        type => "cassandra"
    }
}
output {
   stdout { codec=> rubydebug }
}

日志文件

[qihuang.zheng@dp0652 logstash-1.5.0]$ head /usr/install/cassandra/logs/system.log
ERROR [metrics-graphite-reporter-thread-1] 2016-06-13 18:30:39,502 GraphiteReporter.java:281 - Error sending to Graphite:
java.net.SocketException: 断开的管道
  at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_51]
  at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_51]
  at java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_51]
  at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[na:1.7.0_51]
  at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[na:1.7.0_51]
  at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.7.0_51]
  at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) ~[na:1.7.0_51]
  at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) ~[na:1.7.0_51]

默认一行一个事件，对于有异常的日志文件来说，不经过任何处理肯定是不行的

Using version 0.1.x input plugin 'file'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x codec plugin 'plain'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x output plugin 'stdout'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x codec plugin 'rubydebug'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Registering file input {:path=>["/usr/install/cassandra/logs/system.log"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/home/qihuang.zheng/.sincedb_261f7476b9c9830f1fa5a51db2793e1e", :path=>["/usr/install/cassandra/logs/system.log"], :level=>:info}
Pipeline started {:level=>:info}
Logstash startup completed
{
       "message" => "ERROR [metrics-graphite-reporter-thread-1] 2016-06-13 18:30:39,502 GraphiteReporter.java:281 - Error sending to Graphite:",
      "@version" => "1",
    "@timestamp" => "2016-06-20T08:42:35.543Z",
          "type" => "cassandra",
          "host" => "dp0652",
          "path" => "/usr/install/cassandra/logs/system.log"
}
{
       "message" => "java.net.SocketException: 断开的管道",
      "@version" => "1",
    "@timestamp" => "2016-06-20T08:42:35.544Z",
          "type" => "cassandra",
          "host" => "dp0652",
          "path" => "/usr/install/cassandra/logs/system.log"
}
{
       "message" => "\tat java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_51]",
      "@version" => "1",
    "@timestamp" => "2016-06-20T08:42:35.544Z",
          "type" => "cassandra",
          "host" => "dp0652",
          "path" => "/usr/install/cassandra/logs/system.log"
}

添加多行支持

input {
    file {
        path => "/usr/install/cassandra/logs/system.log"
        start_position => beginning
        type => "cassandra"
        codec => multiline {
          pattern => "^\s"
          what => "previous"
        }
    }
}

将任何以空白开始的行与上一行合并，但是这种方式还是不够理想。实际上下面两条记录应该属于一条

{
    "@timestamp" => "2016-06-20T08:49:54.521Z",
       "message" => "ERROR [metrics-graphite-reporter-thread-1] 2016-06-13 18:30:39,850 GraphiteReporter.java:281 - Error sending to Graphite:",
      "@version" => "1",
          "type" => "cassandra",
          "host" => "dp0652",
          "path" => "/usr/install/cassandra/logs/system.log"
}
{
    "@timestamp" => "2016-06-20T08:49:54.521Z",
       "message" => "java.net.SocketException: 断开的管道\n\tat java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_51]\n\tat java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_51]\n\tat java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_51]\n\tat sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[na:1.7.0_51]\n\tat sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[na:1.7.0_51]\n\tat sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.7.0_51]\n\tat java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) ~[na:1.7.0_51]\n\tat java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) ~[na:1.7.0_51]\n\tat java.io.BufferedWriter.write(BufferedWriter.java:230) ~[na:1.7.0_51]\n\tat java.io.Writer.write(Writer.java:157) ~[na:1.7.0_51]\n\tat com.yammer.metrics.reporting.GraphiteReporter.sendToGraphite(GraphiteReporter.java:271) [metrics-graphite-2.2.0.jar:na]\n\tat com.yammer.metrics.reporting.GraphiteReporter.sendObjToGraphite(GraphiteReporter.java:265) [metrics-graphite-2.2.0.jar:na]\n\tat com.yammer.metrics.reporting.GraphiteReporter.processGauge(GraphiteReporter.java:304) [metrics-graphite-2.2.0.jar:na]\n\tat com.yammer.metrics.reporting.GraphiteReporter.processGauge(GraphiteReporter.java:26) [metrics-graphite-2.2.0.jar:na]\n\tat com.yammer.metrics.core.Gauge.processWith(Gauge.java:28) [metrics-core-2.2.0.jar:na]\n\tat com.yammer.metrics.reporting.GraphiteReporter.printRegularMetrics(GraphiteReporter.java:247) [metrics-graphite-2.2.0.jar:na]\n\tat com.yammer.metrics.reporting.GraphiteReporter.run(GraphiteReporter.java:213) [metrics-graphite-2.2.0.jar:na]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_51]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_51]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_51]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]\n\tat java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "type" => "cassandra",
          "host" => "dp0652",
          "path" => "/usr/install/cassandra/logs/system.log"
}

网上找到的一个Cassandra日志文件的配置：https://github.com/rustyrazorblade/dotfiles/blob/master/logstash.conf

output {
    elasticsearch {
        hosts => ["192.168.6.52:9200"]
        index => "logstash-%{type}-%{+YYYY.MM.dd}"
        document_type => "%{type}"
        workers => 2
        flush_size => 1000
        idle_flush_time => 5
        template_overwrite => true
    }  
    stdout { 
    }
}

input {
    file {
        path => "/usr/install/cassandra/logs/system.log"       
        start_position => beginning
        type => cassandra_system
    }
}

filter {
    if [type] == "cassandra" {
        grok {
            match => {"message" => "%{LOGLEVEL:level}  \[%{WORD:class}:%{NUMBER:line}\] %{TIMESTAMP_ISO8601:timestamp} %{WORD:file}\.java:%{NUMBER:line2} - %{GREEDYDATA:msg}"}
        }
    }
}

性能测试

input { 
  generator { 
    count => 30000000 
  } 
} 
output { 
  stdout { 
    codec => dots 
  } 
  kafka { 
    broker_list => "localhost:9092" 
    topic_id => "test" 
    compression_codec => "snappy" 
  } 
}

bin/logstash agent -f out.conf | pv -Wbart > /dev/null

topic_id => "test" 
compression_codec => "snappy" 
request_required_acks => 1 
serializer_class => "kafka.serializer.StringEncoder" 
request_timeout_ms => 10000 
producer_type => 'async' 
message_send_max_retries => 5 
retry_backoff_ms => 100 
queue_buffering_max_ms => 5000 
queue_buffering_max_messages => 10000 
queue_enqueue_timeout_ms => -1 
batch_num_messages => 1000

Collectd

在52上安装collectd，network插件表示要将当前机器的信息发送到远程服务器10.57.2.26的25856端口

$ sudo yum install collectd
$ sudo mv /etc/collectd.conf /etc/collectd_backup.conf
$ sudo vi /etc/collectd.conf
Hostname "dp0652"
FQDNLookup   true
LoadPlugin interface
LoadPlugin cpu
LoadPlugin memory
LoadPlugin network
LoadPlugin df
LoadPlugin disk
<Plugin interface>
    Interface "eth0"
    IgnoreSelected false
</Plugin>
<Plugin network>
    Server "10.57.2.26" "25826"
</Plugin>
Include "/etc/collectd.d"
$ sudo service collectd start
Starting collectd:                                         [  OK  ]
$ sudo service collectd status
collectd (pid  9295) is running...

开发机器地址是10.57.2.26，监听25826端口，收集来自于collectd发送的信息。

即流程是：在192.168.6.52通过collectd收集系统信息，发送到10.57.2.26的logstash上

$ bin/plugin list | grep collect
logstash-codec-collectd

$ vi collectd.conf
input {
  udp {
    port => 25826
    buffer_size => 1452
    codec => collectd { }
  }
}
output {
  stdout {
    codec => rubydebug 
  }
}
$ bin/logstash -f collectd.conf
Settings: Default pipeline workers: 4
Pipeline main started
{
               "host" => "dp0652",
         "@timestamp" => "2016-05-25T03:49:52.000Z",
             "plugin" => "cpu",
    "plugin_instance" => "21",
      "collectd_type" => "cpu",
      "type_instance" => "system",
              "value" => 1220568,
           "@version" => "1"
}....

ElasticSearch

启动ES

$ cd elasticsearch-2.3.3
$ vi config/elasticsearch.yml
cluster.name: es52
network.host: 192.168.6.52
#discovery.zen.ping.multicast.enabled: false
#http.cors.allow-origin: "/.*/"
#http.cors.enabled: true

$ bin/elasticsearch -d
$ curl http://192.168.6.52:9200/

LogStash+CollectD+ElasticSearch

上面把从collectd搜集到的数据打印到控制台，修改out转存到elasticsearch中。

$ vi elastic.conf
input {
  udp {
    port => 25826
    buffer_size => 1452
    codec => collectd { }
  }
}
output {
    elasticsearch {
        hosts => ["192.168.6.52:9200"]
        index => "logstash-%{type}-%{+YYYY.MM.dd}"
        document_type => "%{type}"
        workers => 2
        flush_size => 1000
        idle_flush_time => 5
        template_overwrite => true
    }
}

$ bin/logstash -f elastic.conf

$ curl http://192.168.6.52:9200/_search?pretty

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 613,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "logstash-%{type}-2016.05.25",
      "_type" : "%{type}",
      "_id" : "AVTmTRv8ihrmowv7fKbo",
      "_score" : 1.0,
      "_source" : {
        "host" : "dp0652",
        "@timestamp" : "2016-05-25T05:04:42.000Z",
        "plugin" : "cpu",
        "plugin_instance" : "22",
        "collectd_type" : "cpu",
        "type_instance" : "steal",
        "value" : 0,
        "@version" : "1"
      }
    }, ......]
  }
}

最终的网络拓扑流程图如下：

实际上collectd是不同的服务器节点（比如Nginx服务器），而logstash和es只需要一台机器即可：

Kiabana

下载后如果ElasticSearch安装在本机，默认直接启动bin/kibana即可

https://www.elastic.co/guide/en/kibana/current/getting-started.html

wget -c https://www.elastic.co/guide/en/kibana/3.0/snippets/shakespeare.json
wget -c https://github.com/bly2k/files/blob/master/accounts.zip?raw=true
wget -c https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz
unzip accounts.zip
gunzip logs.jsonl.gz

shakespeare表结构

{
    "line_id": INT,
    "play_name": "String",
    "speech_number": INT,
    "line_number": "String",
    "speaker": "String",
    "text_entry": "String",
}

建立mapping表结构

curl -XPUT http://192.168.6.52:9200/shakespeare -d '
{
 "mappings" : {
  "_default_" : {
   "properties" : {
    "speaker" : {"type": "string", "index" : "not_analyzed" },
    "play_name" : {"type": "string", "index" : "not_analyzed" },
    "line_id" : { "type" : "integer" },
    "speech_number" : { "type" : "integer" }
   }
  }
 }
}
';

bank表结构

{
    "account_number": INT,
    "balance": INT,
    "firstname": "String",
    "lastname": "String",
    "age": INT,
    "gender": "M or F",
    "address": "String",
    "employer": "String",
    "email": "String",
    "city": "String",
    "state": "String"
}



//logstash-2015.05.18, logstash-2015.05.19, logstash-2015.05.20都要修改
curl -XPUT http://192.168.6.52:9200/logstash-2015.05.20 -d '
{
  "mappings": {
    "log": {
      "properties": {
        "geo": {
          "properties": {
            "coordinates": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}
';

curl -XPOST '192.168.6.52:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -XPOST '192.168.6.52:9200/shakespeare/_bulk?pretty' --data-binary @shakespeare.json
curl -XPOST '192.168.6.52:9200/_bulk?pretty' --data-binary @logs.jsonl

curl '192.168.6.52:9200/_cat/indices?v'

[qihuang.zheng@dp0652 ~]$ curl '192.168.6.52:9200/_cat/indices?v'
health status index                                pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-%{type}-2016.05.25            5   1      12762            0      1.5mb          1.5mb
yellow open   logstash-cassandra_system-2016.06.20   5   1      17125            0      3.9mb          3.9mb
yellow open   logstash-cassandra_system-2016.06.21   5   1        262            0    233.7kb        233.7kb
yellow open   bank                                   5   1       1000            0    442.2kb        442.2kb
yellow open   .kibana                                1   1          5            0     23.3kb         23.3kb
yellow open   shakespeare                            5   1     111396            0     18.4mb         18.4mb
green  open   graylog_0                              1   0       8123            0      2.3mb          2.3mb
yellow open   logstash-2015.05.20                    5   1       4750            0     28.7mb         28.7mb
yellow open   logstash-2015.05.18                    5   1       4631            0     27.4mb         27.4mb
yellow open   logstash-cassandra_system-2016.06.22   5   1         42            0    146.5kb        146.5kb
yellow open   logstash-2015.05.19                    5   1       4624            0     27.8mb         27.8mb

InfluxDB

wget -c https://dl.influxdata.com/influxdb/releases/influxdb-0.13.0_linux_amd64.tar.gz
wget -c https://dl.influxdata.com/telegraf/releases/telegraf-0.13.1_linux_i386.tar.gz
wget -c https://dl.influxdata.com/chronograf/releases/chronograf-0.13.0-1.x86_64.rpm
wget -c https://dl.influxdata.com/kapacitor/releases/kapacitor-0.13.1_linux_amd64.tar.gz
sudo yum localinstall chronograf-0.13.0-1.x86_64.rpm

生成配置文件，启动时指定配置文件

1
2
3

$ cd influxdb-0.13.0-1
$ usr/bin/influxd config > influxdb.generated.conf
$ nohup usr/bin/influxd -config influxdb.generated.conf &

http://192.168.6.52:8083/

用命令行客户端创建数据库，插入数据，查询数据

$ usr/bin/influx
Connected to http://localhost:8086 version 0.13.x
InfluxDB shell 0.13.x
> CREATE DATABASE mydb
> USE mydb
> INSERT cpu,host=serverA,region=us_west value=0.64
> INSERT cpu,host=serverA,region=us_east value=0.45
> INSERT temperature,machine=unit42,type=assembly external=25,internal=37
> SELECT host, region, value FROM cpu
> SELECT * FROM temperature
> SELECT * FROM /.*/ LIMIT 1

在web页面也可以添加数据：

Grafana

grafana-1.9.1

由于Grafana是存静态的，你只需要下载源代码解压，将它部署在Nginx上面就可以了，或者可以用Python的SimpleHTTPServer来跑

1
2
3

$ wget http://grafanarel.s3.amazonaws.com/grafana-1.9.1.tar.gz
$ cd grafana-1.9.1
$ python -m SimpleHTTPServer 8383

http://192.168.6.52:8383

没有任何数据源时，页面是空白的：

添加了数据源后比如influxdb

$ cp config.sample.js config.js
$ vi config.js
datasources: {
  influxdb: {
    type: 'influxdb',
    url: "http://192.168.6.52:8086/db/cassandra-metrics",
    username: 'admin',
    password: 'admin',
  },
  grafana: {
    type: 'influxdb',
    url: "http://192.168.6.52:8086/db/grafana",
    username: 'admin',
    password: 'admin',
    grafanaDB: true
  },
},

重启python进程后，可以看到多了点东西（如果influxdb没有添加admin用户，上面的username和password可以去掉），

但是就是看不到配置相关的按钮，难道是没有权限？而且这个版本进来后，根本没有login页面。

grafana-2.x

https://grafanarel.s3.amazonaws.com/builds/grafana-3.0.3-1463994644.linux-x64.tar.gz

如果用2.5以上的版本包括3.0，和1.9的目录结构相比发生很大变化

[qihuang.zheng@dp0652 grafana-1.9.1]$ tree -L 1
.
├── app
├── build.txt
├── config.js
├── config.sample.js
├── css
├── font
├── img
├── index.html
├── LICENSE.md
├── NOTICE.md
├── plugins
├── README.md
├── test
└── vendor

[qihuang.zheng@dp0652 grafana-3.0.3-1463994644]$ tree -L 2
.
├── bin
│   ├── grafana-cli
│   ├── grafana-cli.md5
│   ├── grafana-server
│   └── grafana-server.md5
├── conf
│   ├── defaults.ini
│   └── sample.ini
├── LICENSE.md
├── NOTICE.md
├── public
│   ├── app
│   ├── css
│   ├── dashboards
│   ├── emails
│   ├── fonts
│   ├── img
│   ├── robots.txt
│   ├── sass
│   ├── test
│   ├── vendor
│   └── views
├── README.md
└── vendor
    └── phantomjs

如果也是用python -m SimpleHTTPServer 8383打开，浏览器不会显示任何东西

参照官方的安装指南分分钟搞定，而且貌似并没有依赖Web服务器之类的。

$ sudo yum install https://grafanarel.s3.amazonaws.com/builds/grafana-3.0.4-1464167696.x86_64.rpm
$ sudo service grafana-server start
Starting Grafana Server: .... FAILED
$ ps -ef|grep grafana
grafana  16078     1  0 13:19 ?        00:00:00 /usr/sbin/grafana-server 
--pidfile=/var/run/grafana-server.pid --config=/etc/grafana/grafana.ini 
cfg:default.paths.data=/var/lib/grafana cfg:default.paths.logs=/var/log/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins
$ vi /var/log/grafana/grafana.log
$ sudo service grafana-server status
grafana-server (pid  16078) is running...

虽然看似失败了，不过日志文件中没有什么错误信息,打开：http://192.168.6.52:3000/
出现了登陆页面，admin/admin

而且grafana的图标是可以点的，也有数据源。首先添加influxdb的数据源

添加一个dashboards①，然后添加一个panel②，在数据源中选择influxdb

默认的查询语句：SELECT mean("value") FROM "measurement" WHERE $timeFilter GROUP BY time($interval) fill(null)

更改为：SELECT mean("value") FROM "cpu" WHERE "host" = "serverA" AND $timeFilter GROUP BY time($interval)

这时候上方会出现图，点击关闭，退出编辑状态。注意不要点击眼睛，一旦变成灰色，表示不发送查询语句
正常q的内容是查询语句：curl -GET 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode "q=SELECT value FROM cpu_load_short WHERE region='us-west'"

往influxdb中插入几条数据：

1 2	cpu,host=serverA,region=us_west value=0.36 cpu,host=serverA,region=us_west value=0.85

在右上角可以选择时间范围，如果时间超过了，没有数据，出现N/A。

Cassandra+InfluxDB+Grafana

使用Grafana监控Cassandra有两个步骤：

将Cassandra监控指标数据收集到InfluxDB中
在Grafana中配置InfluxDB的数据源，展现Cassandra指标数据

参考文档：
http://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-cassandra-2-0-2
https://www.pythian.com/blog/monitoring-cassandra-grafana-influx-db/

其中步骤1有多种方案：

使用Graphite收集数据，发送到InfluxDB中
使用InfluxDB的telegraf输入插件收集数据

Graphite

1）下载metrics-graphite.jar放到Cassandra的lib目录下

2）修改influxdb的配置文件，其中database要对应influxdb中的数据库名称，这里为cassandra-metrics（要在influxdb中手动创建这个数据库）

参考文档中配置文件是config.toml，在新版本中启动influxdb时生成了一个配置文件，实际上是一样的。
旧版本的输入配置项是：[input_plugins.graphite]，新版本的配置项为：[[graphite]]。

[qihuang.zheng@dp0652 influxdb-0.13.0-1]$ vi influxdb.generated.conf
[[graphite]]
  enabled = true
  bind-address = ":2003"
  protocol = "tcp"
  batch-size = 5000
  batch-pending = 10
  batch-timeout = "1s"
  consistency-level = "one"
  separator = "."
  udp-read-buffer = 0
  # database = "graphite"
  database = "cassandra-metrics"
  udp_enabled = true

上面的配置文件表示，InfluxDB将打开2003端口，接收graphite类型的指标数据

3）重启influxdb，使配置文件生效

1 2	$ service influxdb reload # 通过RPM方式安装时，不需要重启 $ influxdb -config influxdb.generated.conf reload

4）在cassandra的conf目录下创建influx的配置文件，其中host指的是influxdb的服务端地址，端口对应上面influxdb.generated.conf的2003端口。
如果influxdb安装在不同节点上，下面的host要指向influxdb的地址。 prefix通常设置为当前机器的IP地址。

[qihuang.zheng@dp0652 conf]$ vi /usr/install/cassandra/conf/influx-reporting.yaml
graphite:
-
  period: 60
  timeunit: 'SECONDS'
  prefix: 'Node1'
  hosts:
  - host: '192.168.6.52'
    port: 2003
  predicate:
    color: "white"
    useQualifiedName: true
    patterns:
    - ".*"

上面的配置文件表示：当前Cassandra节点的每个指标前缀是Node1，都会发送到地址为52的2003端口，即把Cassandra的指标数据发送给InfluxDB。
这里也可以看出指标数据发送的机器只是指定了地址和端口，所以目标地址可以是任何可以接收graphite类型的数据库，不一定是InfluxDB。

5）安装grafana，不需要配置config.js，在web页面也可以添加数据源

6）给cassandra-env.sh添加-D启动选项

1 2	[qihuang.zheng@dp0652 conf]$ vi /usr/install/cassandra/conf/cassandra-env.sh JVM_OPTS="$JVM_OPTS -Dcassandra.metricsReporterConfigFile=influx-reporting.yaml"

7）重启cassandra

1
2

INFO [main] YYYY-MM-DD HH:MM:SS,SSS CassandraDaemon.java:353 - Trying to load metrics-reporter-config from file: influx-reporting.yaml
INFO [main] YYYY-MM-DD HH:MM:SS,SSS GraphiteReporterConfig.java:68 - Enabling GraphiteReporter to 192.168.6.52:2003

8）验证Cassandra指标数据通过metrics-graphite.jar被收集到influxdb中

选择一个measurement，验证有数据被写入：select * FROM "Node1.jvm.daemon_thread_count"

9）在grafana中配置influxdb的数据源，把数据库名称改成cassandra-metrics

配置监控图，修改measurement

10）总结下通过Graphite+InfluxDB收集Cassandra指标数据的步骤流程图：

用正则表达式可以聚合多个节点的指标

1 2	select mean(value) from /.*org.apache.cassandra.metrics.ClientRequest.Read/ select mean(value) from "192.168.6.53.org.apache.cassandra.metrics.ClientRequest.Read.Latency.15MinuteRate"

Telegraph

Hekad

http://hekad.readthedocs.io/en/v0.10.0/index.html

Atlas

https://github.com/Netflix/atlas

Graylog

http://www.cnblogs.com/wjoyxt/p/4961262.html
http://docs.graylog.org/en/2.0/pages/installation/manual_setup.html

准备工作：mongodb安装和启动

mkdir -p /home/qihuang.zheng/data/mongodb
curl -O https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.7.tgz
tar zxf mongodb-linux-x86_64-3.2.7.tgz
nohup mongodb-linux-x86_64-3.2.7/bin/mongod --dbpath /home/qihuang.zheng/data/mongodb & 

sudo rpm -ivh pwgen-2.07-1.el6.x86_64.rpm && sudo yum install perl-Digest-SHA
[qihuang.zheng@dp0652 ~]$ pwgen -N 1 -s 96
XNamqyHbxtV46AHXNMlMAvVeV2dutp2pQaeY9IaOSf9XnwgyYGXqN97SSCQ2OLyR2HR41BtCpxwSMH4kFnr1VHmRNUQvYyic
[qihuang.zheng@dp0652 ~]$ echo -n XNamqyHbxtV46AHXNMlMAvVeV2dutp2pQaeY9IaOSf9XnwgyYGXqN97SSCQ2OLyR2HR41BtCpxwSMH4kFnr1VHmRNUQvYyic | shasum -a 256
cb535aa3ff35e81f69f9014005bcf1ad032048cc123dad735bbf87970eb2cacb  -
[qihuang.zheng@dp0652 ~]$ echo -n admin | shasum -a 256
8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918  -

graylog的配置文件中默认的配置项，单机情况可以不用修改任何配置，不过最好把localhost改成本机IP

$ wget https://fossies.org/linux/misc/graylog-2.0.2.tgz
$ tar zxf graylog-VERSION.tgz && cd graylog
$ vi graylog.conf.example
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret =
root_password_sha2 =
rest_listen_uri = http://127.0.0.1:12900/
#elasticsearch_cluster_name = graylog
#elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300, 127.0.0.2:9500
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
mongodb_uri = mongodb://localhost/graylog

配置文件的路径/etc/graylog/server/server.conf写死在bin/graylogctl脚本中

$ sudo mkdir -p /etc/graylog/server/
$ sudo cp graylog.conf.example /etc/graylog/server/server.conf
$ cat /etc/graylog/server/server.conf |grep -v grep|grep -v ^#|grep -v ^$

password_secret=XNamqyHbxtV46AHXNMlMAvVeV2dutp2pQaeY9IaOSf9XnwgyYGXqN97SSCQ2OLyR2HR41BtCpxwSMH4kFnr1VHmRNUQvYyic
password=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
cluster=`cat ~/elasticsearch-2.3.3/config/elasticsearch.yml |grep cluster.name | awk '{print $2}'`
sed -i -e "s#password_secret =#password_secret = $password_secret#g" server.conf
sed -i -e "s#root_password_sha2 =#root_password_sha2 = $password#g" server.conf
sed -i -e "s#127.0.0.1#192.168.6.52#g" server.conf
sed -i -e "s#localhost#192.168.6.52#g" server.conf
sed -i -e "s#elasticsearch_shards = 4#elasticsearch_shards = 1#g" server.conf
sed -i -e "s#web_listen_uri = http://127.0.0.1:9000/#web_listen_uri = http://192.168.6.52:9999/#g" server.conf
######################################
elasticsearch_cluster_name = $cluster
elasticsearch_discovery_zen_ping_unicast_hosts = 192.168.6.52:9300

graylog的启动脚本在bin/graylogctl，实际上启动命令是java -jar graylog.jar

[qihuang.zheng@dp0652 graylog-2.0.2]$ ll
drwxr-xr-x 2 qihuang.zheng users     4096 6月  20 13:37 bin
-rw-r--r-- 1 qihuang.zheng users    35147 5月  26 23:31 COPYING
drwxr-xr-x 4 qihuang.zheng users     4096 6月  20 12:58 data
-rw-r--r-- 1 qihuang.zheng users    23310 5月  26 23:31 graylog.conf.example
-rw-r--r-- 1 qihuang.zheng users 80950701 5月  26 23:34 graylog.jar    ⬅️
drwxr-xr-x 3 qihuang.zheng users     4096 6月  20 11:55 lib
drwxr-xr-x 2 qihuang.zheng users     4096 6月  20 13:02 log
drwxr-xr-x 2 qihuang.zheng users     4096 5月  26 23:33 plugin

如果要自定义log配置，修改bin/graylogctl的start部分在-jar前添加配置文件路径

1	-Dlog4j.configurationFile=file:///home/qihuang.zheng/graylog-2.0.2/log4j2.xml -jar "${GRAYLOG_SERVER_JAR}" server

启动graylog，同时会启动web服务，默认端口是9000，不过和HDFS重了，所以上面把web_listen_uri修改成9999

[qihuang.zheng@dp0652 ~]$ sudo graylog-2.0.2/bin/graylogctl start
Starting graylog-server ...
[qihuang.zheng@dp0652 ~]$ sudo graylog-2.0.2/bin/graylogctl status
graylog-server running with PID 8600
[qihuang.zheng@dp0652 ~]$ ll /etc/graylog/server/
-rw-r--r-- 1 root root    36 6月  20 12:58 node-id
-rw-r--r-- 1 root root 23496 6月  20 12:57 server.conf
[qihuang.zheng@dp0652 ~]$ cat /etc/graylog/server/node-id
eb943e44-3464-47be-9c07-2a554de71428
[qihuang.zheng@dp0652 graylog-2.0.2]$ ps -ef|grep graylog
root     33748     1 48 14:17 pts/0    00:01:20 /home/qihuang.zheng/jdk1.8.0_91/bin/java -Djava.library.path=bin/../lib/sigar -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -jar graylog.jar server -f /etc/graylog/server/server.conf -p /tmp/graylog.pid
506      38283 29183  0 14:20 pts/0    00:00:00 grep graylog
[qihuang.zheng@dp0652 graylog-2.0.2]$ cat /tmp/graylog.pid
33748

日志文件

[qihuang.zheng@dp0652 graylog-2.0.2]$ cat log/graylog-server.log
2016-06-20 14:17:46,602 INFO : kafka.log.LogManager - Loading logs.
2016-06-20 14:17:46,662 INFO : kafka.log.LogManager - Logs loading complete.
2016-06-20 14:17:46,662 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at data/journal
2016-06-20 14:17:46,676 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2016-06-20 14:17:46,706 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[192.168.6.52:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2016-06-20 14:17:46,731 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, all=[ServerDescription{address=192.168.6.52:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-06-20 14:17:46,752 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:40}] to 192.168.6.52:27017
2016-06-20 14:17:46,753 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=192.168.6.52:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 7]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=512946}
2016-06-20 14:17:46,758 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:41}] to 192.168.6.52:27017
2016-06-20 14:17:46,929 INFO : org.graylog2.plugin.system.NodeId - Node ID: eb943e44-3464-47be-9c07-2a554de71428
2016-06-20 14:17:46,978 INFO : org.elasticsearch.node - [graylog-eb943e44-3464-47be-9c07-2a554de71428] version[2.3.2], pid[33748], build[b9e4a6a/2016-04-21T16:03:47Z]
2016-06-20 14:17:46,978 INFO : org.elasticsearch.node - [graylog-eb943e44-3464-47be-9c07-2a554de71428] initializing ...
2016-06-20 14:17:46,984 INFO : org.elasticsearch.plugins - [graylog-eb943e44-3464-47be-9c07-2a554de71428] modules [], plugins [graylog-monitor], sites []
2016-06-20 14:17:48,243 INFO : org.elasticsearch.node - [graylog-eb943e44-3464-47be-9c07-2a554de71428] initialized
2016-06-20 14:17:48,304 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.4.Final
2016-06-20 14:17:48,419 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-06-20 14:17:49,918 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2016-06-20 14:17:50,510 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.0.2 (4da1379) starting up
2016-06-20 14:17:50,514 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2016-06-20 14:17:50,532 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 24 periodicals ...
2016-06-20 14:17:50,538 INFO : org.elasticsearch.node - [graylog-eb943e44-3464-47be-9c07-2a554de71428] starting ...
2016-06-20 14:17:50,544 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
2016-06-20 14:17:50,546 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:43}] to 192.168.6.52:27017
2016-06-20 14:17:50,546 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:42}] to 192.168.6.52:27017
2016-06-20 14:17:50,553 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
2016-06-20 14:17:50,573 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2016-06-20 14:17:50,648 INFO : org.elasticsearch.transport - [graylog-eb943e44-3464-47be-9c07-2a554de71428] publish_address {127.0.0.1:9350}, bound_addresses {[::1]:9350}, {127.0.0.1:9350}
2016-06-20 14:17:50,654 INFO : org.elasticsearch.discovery - [graylog-eb943e44-3464-47be-9c07-2a554de71428] es52/rWpoduohQ1CXcyoAuwzNtg
2016-06-20 14:17:53,239 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [192.168.6.52:9999]
2016-06-20 14:17:53,241 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2016-06-20 14:17:53,242 INFO : org.graylog2.initializers.WebInterfaceService - Started Web Interface at <http://192.168.6.52:9999/>
2016-06-20 14:17:53,657 WARN : org.elasticsearch.discovery - [graylog-eb943e44-3464-47be-9c07-2a554de71428] waited for 3s and no initial state was set by the discovery
2016-06-20 14:17:53,657 INFO : org.elasticsearch.node - [graylog-eb943e44-3464-47be-9c07-2a554de71428] started
2016-06-20 14:17:53,730 INFO : org.elasticsearch.cluster.service - [graylog-eb943e44-3464-47be-9c07-2a554de71428] detected_master {Daisy Johnson}{ZVzCrnWoRsKVnRryYLE6BQ}{192.168.6.52}{192.168.6.52:9300}, added {{Daisy Johnson}{ZVzCrnWoRsKVnRryYLE6BQ}{192.168.6.52}{192.168.6.52:9300},}, reason: zen-disco-receive(from master [{Daisy Johnson}{ZVzCrnWoRsKVnRryYLE6BQ}{192.168.6.52}{192.168.6.52:9300}])
2016-06-20 14:17:56,443 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [192.168.6.52:12900]
2016-06-20 14:17:56,443 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer-1] Started.
2016-06-20 14:17:56,444 INFO : org.graylog2.shared.initializers.RestApiService - Started REST API at <http://192.168.6.52:12900/>
2016-06-20 14:17:56,445 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2016-06-20 14:17:56,446 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2016-06-20 14:17:56,446 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {JournalReader [RUNNING]=1, BufferSynchronizerService [RUNNING]=1, OutputSetupService [RUNNING]=1, InputSetupService [RUNNING]=2, MetricsReporterService [RUNNING]=2, KafkaJournal [RUNNING]=4, PeriodicalsService [RUNNING]=51, WebInterfaceService [RUNNING]=2705, IndexerSetupService [RUNNING]=3211, RestApiService [RUNNING]=5912}
2016-06-20 14:17:56,451 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2016-06-20 14:18:00,548 INFO : org.graylog2.indexer.Deflector - Did not find an deflector alias. Setting one up now.
2016-06-20 14:18:00,552 INFO : org.graylog2.indexer.Deflector - There is no index target to point to. Creating one now.
2016-06-20 14:18:00,554 INFO : org.graylog2.indexer.Deflector - Cycling deflector to next index now.
2016-06-20 14:18:00,555 INFO : org.graylog2.indexer.Deflector - Cycling from <none> to <graylog_0>
2016-06-20 14:18:00,555 INFO : org.graylog2.indexer.Deflector - Creating index target <graylog_0>...
2016-06-20 14:18:00,614 INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
2016-06-20 14:18:00,698 INFO : org.graylog2.indexer.Deflector - Waiting for index allocation of <graylog_0>
2016-06-20 14:18:00,800 INFO : org.graylog2.indexer.Deflector - Done!
2016-06-20 14:18:00,800 INFO : org.graylog2.indexer.Deflector - Pointing deflector to new target index....
2016-06-20 14:18:00,845 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <bd3c64c0-36ae-11e6-bcbc-02423384d6ab> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2016-06-20 14:18:00,845 INFO : org.graylog2.indexer.Deflector - Done!
2016-06-20 14:18:00,845 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Calculating ranges for index graylog_0.
2016-06-20 14:18:00,964 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_0] in [117ms].
2016-06-20 14:18:00,973 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Created ranges for index graylog_0.
2016-06-20 14:18:00,973 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <bd3c64c0-36ae-11e6-bcbc-02423384d6ab> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 128ms.

查看ES是否创建索引

1
2
3

[qihuang.zheng@dp0652 graylog-2.0.2]$ curl http://192.168.6.52:9200/_cat/indices
yellow open logstash-%{type}-2016.05.25 5 1 12762 0 1.5mb 1.5mb
green  open graylog_0                   1 0     0 0  159b  159b

查看MongoDB是否创建数据库

[qihuang.zheng@dp0652 graylog-2.0.2]$ ~/mongodb-linux-x86_64-3.2.7/bin/mongo
MongoDB shell version: 3.2.7
connecting to: test
> show dbs;
graylog  0.001GB
local    0.000GB
test     0.000GB
> show collections
cluster_config
cluster_events
collectors
content_packs
grok_patterns
index_failures
index_ranges
nodes
notifications
pipeline_processor_pipelines
pipeline_processor_pipelines_streams
pipeline_processor_rules
roles
sessions
system_messages
users
> db.nodes.count()
1
> db.cluster_config.count()
11
> db.nodes.find()
{ "_id" : ObjectId("5767780ab01412219843147b"), "is_master" : true, "hostname" : "dp0652", "last_seen" : 1466404508, "transport_address" : "http://192.168.6.52:12900/", "type" : "SERVER", "node_id" : "eb943e44-3464-47be-9c07-2a554de71428" }
> db.cluster_config.distinct("type")
[
  "org.graylog2.bundles.ContentPackLoaderConfig",
  "org.graylog2.cluster.UserPermissionMigrationState",
  "org.graylog2.indexer.management.IndexManagementConfig",
  "org.graylog2.indexer.retention.strategies.ClosingRetentionStrategyConfig",
  "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig",
  "org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategyConfig",
  "org.graylog2.indexer.rotation.strategies.SizeBasedRotationStrategyConfig",
  "org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig",
  "org.graylog2.indexer.searches.SearchesClusterConfig",
  "org.graylog2.periodical.IndexRangesMigrationPeriodical.MongoIndexRangesMigrationComplete",
  "org.graylog2.plugin.cluster.ClusterId"
]

注意必须修改server.conf的elasticsearch_cluster_name配置项和已经安装的elasticsearch的名称一样，否则会报错：

2016-06-20 14:06:47,920 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
org.elasticsearch.discovery.MasterNotDiscoveredException
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226) ~[graylog.jar:?]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236) ~[graylog.jar:?]
        at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804) ~[graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

用admin/admin登陆：http://192.168.6.52:9999，密码是echo -n admin | shasum -a 256中的admin

发送数据到gralog

UDP

1.在web页面的System/Inputs添加一个RAW UDP文本

2.模拟向端口输入数据

1	[qihuang.zheng@dp0652 graylog-2.0.2]$ echo "Hello Graylog, let's be friends." \| nc -w 1 -u 127.0.0.1 5555

3.在Search可以查询到这条消息

GELF

1.配置gelf

2.发送gelf数据

curl -XPOST http://192.168.6.52:12201/gelf -p0 -d ‘{“short_message”:”Hello there”, “host”:”example.org”, “facility”:”test”, “_foo”:”bar”}’

Logstash

Input配置为GELF UDP，端口为12202，和GELF HTTP的12201区分开来

1 2	➜ logstash-2.3.2 bin/logstash -e 'input { stdin {} } output { gelf {host => "192.168.6.52" port => 12202 } }' logstash message

syslog

kafka

1.创建topic

1
2
3

bin/kafka-topics.sh --create --zookeeper 192.168.6.55:2181,192.168.6.56:2181,192.168.6.57:2181 --replication-factor 1 --partitions 1 --topic graylog-test
bin/kafka-console-producer.sh --broker-list 192.168.6.55:9092,192.168.6.56:9092,192.168.6.57:9092 --topic graylog-test
hello msg from kafka

2.配置kafka输入

3.查看收集的日志

Graylog-Cassandra

https://github.com/Graylog2/graylog-plugin-metrics-reporter

Ref

分享到：

mysql 分区表 partion | 使用 Kapacitor 对 InfluxDB 数据进行统计 ...

2016-07-23 21:06
浏览 1601
评论(0)
分类:开源软件
查看更多

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论

elk

日志监控系统

ELK

LogStash

标准I/O

File Input

性能测试

Collectd

ElasticSearch

LogStash+CollectD+ElasticSearch

Kiabana

InfluxDB

Grafana

grafana-1.9.1

grafana-2.x

Cassandra+InfluxDB+Grafana

Graphite

Telegraph

Hekad

Atlas

Graylog

发送数据到gralog

UDP

GELF

Logstash

syslog

kafka

Graylog-Cassandra

Ref

评论

发表评论

相关推荐

最近访客 更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论

elk

日志监控系统

ELK

LogStash

标准I/O

File Input

性能测试

Collectd

ElasticSearch

LogStash+CollectD+ElasticSearch

Kiabana

InfluxDB

Grafana

grafana-1.9.1

grafana-2.x

Cassandra+InfluxDB+Grafana

Graphite

Telegraph

Hekad

Atlas

Graylog

发送数据到gralog

UDP

GELF

Logstash

syslog

kafka

Graylog-Cassandra

Ref

评论

发表评论

相关推荐

python 类装饰器的四种场景写法

python logging 重复写日志问题

CSS 黑魔法小技巧，让你少写不必要的JS，代码更优雅

Redis 数据类型及应用场景

数据可视化的开源方案: Superset vs Redash vs Metabase

Nginx 限制单个IP的并发连接数/速度来减缓垃圾蜘蛛爬虫采集

docker 升级到最新版

PyMongo 常见问题

Python实战mongodb第3篇: Pymongo的分页查询

SQL转 MongoDB语法速查表

Logstash由于时区导致8小时时差解决方案

Kombu 源码解析一

分布式SQL查询引擎Presto原理介绍

一行Python代码实现树结构

命令行神器 Click

django 流式大文件文件下载

filebeat to elasticsearch针对于filebeat端性能优化--性能提升230%

re必杀技正则表达式大全——包括校验数字、字符、一些特殊的需求等等

n种elasticsearch按照日期定时批量删除索引

10个Python面试常问的问题

最近访客更多访客>>