个人博客

Nginx json日志格式采集(nginx->filebeat->logstash->elasticsearch)①

25 02月
作者:西洪室|分类:技术

部署服务器:

Nginx服务器IP                192.168.12.200

Elasticsearch服务器IP     192.168.16.247

    1、修改nginx.conf配置文件,添加json日志格式:

   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
   log_format json escape=json  '{"@timestamp":"$time_iso8601",'
                    '"@source":"$server_addr",'
                    '"hostname":"$hostname",'
                    '"ip":"$http_x_forwarded_for",'
                    '"client":"$remote_addr",'
                    '"request_method":"$request_method",'
                    '"scheme":"$scheme",'
                    '"domain":"$server_name",'
                    '"client_host":"$host",'
                    '"referer":"$http_referer",'
                    '"request":"$request_uri",'
                    '"args":"$args",'
                    '"size":$body_bytes_sent,'
                    '"status": $status,'
                    '"responsetime":$request_time,'
                    '"upstreamtime":"$upstream_response_time",'
                    '"upstreamaddr":"$upstream_addr",'
                    '"http_user_agent":"$http_user_agent",'
                    '"https":"$https"'
                    '}';

注:escape=json 参数,在配置日志格式时加上此参数可以不转义变量内容

日志格式

{"@timestamp":"2019-01-31T11:26:48+08:00","@source":"192.168.12.200","hostname":"proxy.server","ip":"","client":"183.3.239.170","request_method":"GET","scheme":"http","domain":"www.51hangyu.com","referer":"http://www.51hangyu.com/chat","request":"/static/static/css/main.97f6b853.css","args":"","size":460413,"status": 200,"responsetime":0.108,"upstreamtime":"0.002","upstreamaddr":"192.168.12.15:80","http_user_agent":"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)","https":""}

    2、filebeat配置

filebeat.yml需要修改的三个地方:

    2-1. 读取日志路径和分类

- type: log
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  fields:
     nginx: nginx-access

    ① 引入字段fields,定义nginx: nginxaccess,在logstash里用if [fields][nginx] == "nginxaccess" 引用进行区分

    ② json.keys_under_root: 默认这个值是FALSE的,也就是我们的json日志解析后会被放在json键上。设为TRUE,所有的keys就会被放到根节点

    ③ json.overwrite_keys: 是否要覆盖原有的key,这是关键配置,将keys_under_root设为TRUE后,再将overwrite_keys也设为TRUE,就能把filebeat默认的key值给覆盖了


    2-2. filebeat模块文件路径

#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: /etc/filebeat/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: false
    2-3. 日志输出目的(logstash)
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.16.246:5044"]

    完成后重启filebeat

systemctl restart filebeat

    3、logstash配置

    3-1. 在/etc/logstash/conf.d/下建立nginx输入、过滤、输出配置文件

[root@logstash ~]# vim /etc/logstash/conf.d/nginx.conf
input {
    beats {
      port => 5044
      codec => json
      client_inactivity_timeout => 600
    }
}
filter {
   if [fields][nginx] == "nginx-access" {
       mutate {
           convert => [ "status","integer" ]
           convert => [ "size","integer" ]
           convert => [ "upstreatime","float" ]
       }
       date {
           match => [ "timestamp" ,"dd/MMM/yyyy:HH:mm:ss Z" ]
           target => "@timestamp"
       }
       mutate {
           remove_field => "timestamp"
       }
       geoip {
           source => "client"
           target => "geoip"
           database => "/opt/geolite/GeoLite2-City.mmdb"
           add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
           add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
       }
       mutate {
           convert => ["[geoip][coordinates]", "float"]
           remove_field => "message"
           remove_field => "host"
       }
       if "_geoip_lookup_failure" in [tags] { drop { } }
    }
}
output {
   if [fields][nginx] == "nginx-access" {
       elasticsearch {
       hosts => ["http://192.168.16.247:9200"]
       index => "logstash-%{[fields][nginx]}-%{+YYYY.MM.dd}"
       }
    }
}

注:

    ① client_inactivity_timeout => 600 默认tcp超时关闭时间60秒,filebeat数据不会漏传,但日志会提示错误,然后自动重新连接;修改成600秒后,filebeat就不提示错误了。

    ② date: 时间处理,该插件很实用,主要是用你日志文件中事件的事件来对timestamp进行转换,导入老的数据必备!

        match:匹配到timestamp字段后,修改格式为dd/MMM/yyyy:HH:mm:ss Z (2018-11-26T22:11:47.000Z)

    ③ mutate:数据修改

        remove_field: 移除timestamp字段。

        修改了nginx配置文件,不进行转义,下面的替换可以不使用

        gsub => [ 
            "referer", "\\x22", '"' 空格替换成 "
            "referer", "\\x0A", "\n" 替换成换行
            "http_user_agent", "\x09", " " 将\x09转义为空格
             ]

    ④ if "_geoip_lookup_failure" in [tags] { drop { } } ### 如果解析的地址是内网IP geoip解析将会失败,会生成_geoip_lookup_failure字段,这段话的意思是如果内网地址 drop掉这个字段。

    ⑤ index => "logstash-%{[fields][nginx]}-%{+YYYY.MM.dd}" 在kibana里使用世界地图,index需要前缀logstash-*,匹配geo_point类型

    注:因为elasticsearch的自带的模板(elasticsearch-template-es6x.json)是以"template" : "logstash-*",开头的,如果index => "logstash_kibana_nginx",会有如下提示

提示: No Compatible Fields: The "logstash_kibana_nginx" index pattern does not contain any of the following field types: geo_point 没有相应"geo_point"类型

附默认模板信息

[root@logstash ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/elasticsearch-template-es6x.json
{
  "template" : "logstash-*",
  "version" : 60001,
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
      "dynamic_templates" : [ {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text",
            "norms" : false
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text", "norms" : false,
            "fields" : {
              "keyword" : { "type": "keyword", "ignore_above": 256 }
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date"},
        "@version": { "type": "keyword"},
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }
      }
    }
  }
}


    3-2. 创建自定义grok规则文件(目前没用上)

[root@logstash ~]# mkdir /etc/logstash/patterns/
[root@logstash ~]# chown -R logstash: /etc/logstash/patterns/
[root@logstash ~]# vim /etc/logstash/patterns/nginx
NGINXACCESS %{IPORHOST:clientip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)") %{QS:agent} %{QS:x_forwarded_for}
NGINXERROR %{WORD:err_info}


浏览3575 评论0
返回
目录
返回
首页
简单Mysql状态监控(zabbix默认mysql监控模板) Nginx日志展示(elasticsearch->kibana)②

发表评论