ElasticSearch, Kibana, and Fluentd 的简单部署实例

参考：http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd

要求：JDK 1.6及以上，ruby 1.9.2及以上(最后附安装过程mac os)



一、安装ElasticSearch：
1、下载：http://www.elasticsearch.org/download/，本人下载的是：elasticsearch-0.90.3.tar.gz 包

$ tar zxvf elasticsearch-0.90.0.RC2.tar.gz
$ cd elasticsearch-0.90.0.RC2/

Once installation is complete, start ElasticSearch.

$ ./bin/elasticsearch -f

2、安装Kibana(http://kibana.org/intro.html)

Extract your archive
Open KibanaConfig.rb in your favorite editor
Set Elasticsearch = "localhost:9200" to your ElasticSearch server

Run gem install bundler
Run bundle install

运行

$ bundle exec ruby kibana.rb

3、fluentd的安装，参照：http://docs.fluentd.org/categories/installation

4、elasticsearch plugin 的安装：https://github.com/uken/fluent-plugin-elasticsearch(运行时如有问题，请查阅：https://gist.github.com/y-matsuwitter/4951605)
如下：
# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql
# vi /etc/td-agent/td-agent.conf
file1.txt
# service td-agent restart
Starting td-agent: 2013-02-14 18:14:40 +0900: fluent/supervisor.rb:187:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'mysql'. Run 'gem search -rd fluent-plugin' to find plugins"
                                                           [FAILED]
file2.txt
# /usr/lib64/fluent/ruby/bin/gem environment
~~~~~中略~~~~~~~~~~~~~
  - GEM PATHS:
     - /usr/local/rvm/gems/ruby-1.9.3-p327
     - /usr/local/rvm/gems/ruby-1.9.3-p327@global
~~~~~中略~~~~~~~~~~~~~
file3.txt
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems
file4.txt

GEM_HOME="/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/"
GEM_PATH="/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/"
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql


-----------------------------------------------------
三个节点：192.168.1.4，192.168.1.5，192.168.1.6；

fluentd的配置，
 
192.168.1.4
******************************************************************** 
 
  <source>
    type http
    port 8888
  </source>
 
  <source>
    type forward
    port 24224
    bind 0.0.0.0
  </source>
 
  <source>
    type tail
    format /^ *(?<level>[^ ]* )(?<type>[^ ]* )(?<time>[^ ]* [^ ]* )(?<message>.*)$/
    path /data/cassandra/log/system.log
    pos_file /root/logserver/fluentd/logpos/system.pos
    tag 192.168.1.4
  </source>
 
  <match 192.**>
  type copy
  <store>
   type file
   path /root/logserver/fluentd/log/cassandra
  </store>
  <store>
   type elasticsearch
   logstash_format true
   flush_interval 10s
        include_tag_key true
        tag_key host
  </store>
  </match>
 
  <match debug.**>
    type stdout
  </match>
 
 
192.168.1.5/6 
********************************************************************
  <source>
    type http
    port 8888
  </source>
  <source>
    type tail
    format /^ *(?<level>[^ ]* )(?<type>[^ ]* )(?<time>[^ ]* [^ ]* )(?<message>.*)$/
    path /data/cassandra/log/system.log
    pos_file /root/logserver/fluentd/logpos/system.pos
    tag 192.168.1.5
  </source>
  <match debug.**>
    type stdout
  </match>
  <match 192.**>
    type forward
    send_timeout 60s
    recover_wait 10s
    heartbeat_interval 10s
    phi_threshold 8
    hard_timeout 60s
    <server>
      name icecrown
      host 192.168.1.4
      port 24224
      weight 60
    </server>
    <secondary>
      type file
      path /root/logserver/fluentd/forward-failed
    </secondary>
  </match>
  ********************************************************************
--------------ok-------------------------------------



安装相关资料
ruby install:
1、rvm:的安装（http://stackoverflow.com/questions/11677771/rvm-command-not-found-mac-ox）

curl -L https://get.rvm.io | bash -s -- --version latest

$ source ~/.rvm/scripts/rvm

then

$ type rvm | head -n 1

If the output is:

rvm is a function

You may need to add "source ~/.rvm/scripts/rvm" to your ~/.bash_profile file
或者（this worked for me):

rm -rf ~/.rvm
curl -L https://get.rvm.io | bash -s stable

use rvm install ruby(http://misheska.com/blog/2013/06/16/using-rvm-to-manage-multiple-versions-of-ruby/)

$ rvm install 1.9.3

You may get an error message saying “There was an error while trying to resolve rubygems version for ‘latest’. Halting the installation.” Just run the install again like so to fix the issue:

$ rvm reinstall 1.9.3

Verify the RVM install by running the following commands:

$ rvm -h
$ rvm list
$ rvm use 1.9.3
$ rvm rubygems latest

To ensure that the newer Ruby 1.9.3 is used by default instead of the system 1.8.7 version, run the following command:

$ rvm use 1.9.3 --default

linux 上的安装：http://tecadmin.net/how-to-install-ruby-2-0-0-on-centos-6-using-rvm/#