logstash-2.3.1按日产生索引(%{+YYYY.MM.dd})产生时间比预计晚8小时问题

logstash-2.3.1按日产生索引(%{+YYYY.MM.dd})产生时间比预计晚8小时问题

 

由于Elasticsearch、Logstash内部，对时间类型字段，是统一采用 UTC 时间，outputs/elasticsearch中常用的 %{+YYYY.MM.dd}

这种写法必须读取 @timestamp，为了解决索引产生的时间问题，必须先解决@timestamp时区问题。结合网上资料有以下几种思路：

 

一：使用filter(目前使用)

filter{

ruby {

code => "event.timestamp.time.localtime"

}

}

或

filter{

ruby {

code => "event['@timestamp'] = LogStash::Timestamp.coerce(event['@timestamp'].time.localtime)"

}

}

@timestamp将采用系统当前时间

 

二：如果需要将日志文件中的日期映射成@timestamp，可以使用以下类似配置

filter{

date {

match => ["logdate", "yyyyMMddHHmmssSSS"]

locale => "cn"

timezone => "Asia/Shanghai"

}

}

其中logdate为需要被映射的字段，且数据格式是：yyyyMMddHHmmssSSS，如果无该字段或格式错误，将采用系统默认时间代替。

timezone也可以写成"+08:00",详见：http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html

如果需要删除logdate字段可以使用filter的mutate，配置如下

filter{

mutate {

remove_field => ["logdate"]

}

}

这样索引中将只保留@timestamp字段。

三：修改logstash相关配置文件

老版本（1.5之前的版本）：

WithZone (org.Joda.Time.DateTimeZone::UTC)

修改为：

WithZone (org.Joda.Time.DateTimeZone.getDefault())

 

1.5以后的版本：

路径：/vendor/bundle/jruby/1.9/gems/logstash-core-(version)-java/lib/logstash/event.rb

我的例子（logstash-2.3.1）：

vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/timestamp.rb

63行

UTC = org.joda.time.DateTimeZone.forID(“UTC”)

修改为：

UTC = org.joda.time.DateTimeZone.getDefault()

 

解决了索引中@timestamp的问题后，依然不可以解决索引(%{+YYYY.MM.dd})产生时间比预计晚8小时问题，因为%{+YYYY.MM.dd}对应日期

在UTC时依然比时间情况晚了8小时，为了解决该问题，可以使用以下方式：

一：如果使用日期映射，最简单的方法是使日志文件中的日期+8个小时

二：使用filter

filter{

ruby {

code => "event['@timestamp'] = LogStash::Timestamp.coerce(event['@timestamp'].time.localtime + 8*60*60)"

}

}

这两种方法对于那些仅通过索引计算，且不需要通过kibana等查询展示的是可以，否则， 会出现Elasticsearch 原有的 ["now-1h" TO "now"]

这种方便的搜索语句无法正常使用的尴尬，不推荐使用。

 

三:修改配置

logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/string_interpolation.rb

139行.withZone(org.joda.time.DateTimeZone::UTC)修改成.withZone(org.joda.time.DateTimeZone.forID('Asia/Shanghai')) 或 .withLocale(java.util.Locale::CHINA)

 

至于如何定位到string_interpolation.rb文件

1：使用filter的mutate删除@timestamp字段

filter{

mutate {

remove_field => ["@timestamp"]

}

}

2：重启配置会出现如下堆栈错误

LogStash::Error: Unable to format in string "YYYY.MM.dd", @timestamp field not found

evaluate at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/string_interpolation.rb:145

evaluate at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/string_interpolation.rb:90

collect at org/jruby/RubyArray.java:2409

evaluate at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/string_interpolation.rb:90

evaluate at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/string_interpolation.rb:27

sprintf at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.1-java/lib/logstash/event.rb:199

event_action_params at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:131

event_action_tuple at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:35

multi_receive at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:29

map at org/jruby/RubyArray.java:2414

multi_receive at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:29

each_slice at org/jruby/RubyArray.java:1653

multi_receive at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:28

worker_multi_receive at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb:130

multi_receive at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb:114

output_batch at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:301

each at org/jruby/RubyHash.java:1342

output_batch at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:301

worker_loop at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:232

start_workers at /home/cloud/tools/ELK/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:201