Nginx json日志格式采集（nginx->filebeat->logstash->elasticsearch）①
部署服务器：
Nginx服务器IP 192.168.12.200
Elasticsearch服务器IP 192.168.16.247
1、修改nginx.conf配置文件，添加json日志格式：
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format json escape=json '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",'
'"hostname":"$hostname",'
'"ip":"$http_x_forwarded_for",'
'"client":"$remote_addr",'
'"request_method":"$request_method",'
'"scheme":"$scheme",'
'"domain":"$server_name",'
'"client_host":"$host",'
'"referer":"$http_referer",'
'"request":"$request_uri",'
'"args":"$args",'
'"size":$body_bytes_sent,'
'"status": $status,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamaddr":"$upstream_addr",'
'"http_user_agent":"$http_user_agent",'
'"https":"$https"'
'}';注：escape=json 参数，在配置日志格式时加上此参数可以不转义变量内容
日志格式
{"@timestamp":"2019-01-31T11:26:48+08:00","@source":"192.168.12.200","hostname":"proxy.server","ip":"","client":"183.3.239.170","request_method":"GET","scheme":"http","domain":"www.51hangyu.com","referer":"http://www.51hangyu.com/chat","request":"/static/static/css/main.97f6b853.css","args":"","size":460413,"status": 200,"responsetime":0.108,"upstreamtime":"0.002","upstreamaddr":"192.168.12.15:80","http_user_agent":"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)","https":""}2、filebeat配置
filebeat.yml需要修改的三个地方：
2-1. 读取日志路径和分类
- type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true fields: nginx: nginx-access
① 引入字段fields，定义nginx: nginxaccess，在logstash里用if [fields][nginx] == "nginxaccess" 引用进行区分
② json.keys_under_root： 默认这个值是FALSE的，也就是我们的json日志解析后会被放在json键上。设为TRUE，所有的keys就会被放到根节点
③ json.overwrite_keys: 是否要覆盖原有的key，这是关键配置，将keys_under_root设为TRUE后，再将overwrite_keys也设为TRUE，就能把filebeat默认的key值给覆盖了
2-2. filebeat模块文件路径
#============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: /etc/filebeat/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false 2-3. 日志输出目的(logstash) #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["192.168.16.246:5044"]
完成后重启filebeat
systemctl restart filebeat
3、logstash配置
3-1. 在/etc/logstash/conf.d/下建立nginx输入、过滤、输出配置文件
[root@logstash ~]# vim /etc/logstash/conf.d/nginx.conf
input {
beats {
port => 5044
codec => json
client_inactivity_timeout => 600
}
}
filter {
if [fields][nginx] == "nginx-access" {
mutate {
convert => [ "status","integer" ]
convert => [ "size","integer" ]
convert => [ "upstreatime","float" ]
}
date {
match => [ "timestamp" ,"dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
}
mutate {
remove_field => "timestamp"
}
geoip {
source => "client"
target => "geoip"
database => "/opt/geolite/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
}
mutate {
convert => ["[geoip][coordinates]", "float"]
remove_field => "message"
remove_field => "host"
}
if "_geoip_lookup_failure" in [tags] { drop { } }
}
}
output {
if [fields][nginx] == "nginx-access" {
elasticsearch {
hosts => ["http://192.168.16.247:9200"]
index => "logstash-%{[fields][nginx]}-%{+YYYY.MM.dd}"
}
}
}注：
① client_inactivity_timeout => 600 默认tcp超时关闭时间60秒，filebeat数据不会漏传，但日志会提示错误，然后自动重新连接；修改成600秒后，filebeat就不提示错误了。
② date: 时间处理，该插件很实用，主要是用你日志文件中事件的事件来对timestamp进行转换，导入老的数据必备！
match：匹配到timestamp字段后，修改格式为dd/MMM/yyyy:HH:mm:ss Z (2018-11-26T22:11:47.000Z)
③ mutate：数据修改
remove_field： 移除timestamp字段。
修改了nginx配置文件，不进行转义，下面的替换可以不使用
gsub => [ "referer", "\\x22", '"' 空格替换成 " "referer", "\\x0A", "\n" 替换成换行 "http_user_agent", "\x09", " " 将\x09转义为空格 ]
④ if "_geoip_lookup_failure" in [tags] { drop { } } ### 如果解析的地址是内网IP geoip解析将会失败，会生成_geoip_lookup_failure字段，这段话的意思是如果内网地址 drop掉这个字段。
⑤ index => "logstash-%{[fields][nginx]}-%{+YYYY.MM.dd}" 在kibana里使用世界地图，index需要前缀logstash-*，匹配geo_point类型
注：因为elasticsearch的自带的模板（elasticsearch-template-es6x.json）是以"template" : "logstash-*",开头的，如果index => "logstash_kibana_nginx",会有如下提示
提示： No Compatible Fields: The "logstash_kibana_nginx" index pattern does not contain any of the following field types: geo_point 没有相应"geo_point"类型
附默认模板信息：
[root@logstash ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/elasticsearch-template-es6x.json
{
"template" : "logstash-*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}3-2. 创建自定义grok规则文件（目前没用上）
[root@logstash ~]# mkdir /etc/logstash/patterns/
[root@logstash ~]# chown -R logstash: /etc/logstash/patterns/
[root@logstash ~]# vim /etc/logstash/patterns/nginx
NGINXACCESS %{IPORHOST:clientip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)") %{QS:agent} %{QS:x_forwarded_for}
NGINXERROR %{WORD:err_info}目录 返回
首页