graylog实现日志监控

graylog

graylog是一个轻量级的日志管理工具,依托elasticsearch作为日志存储中间件,MongoDB作为元数据信息存储中间件.自带WEB-UI界面,LDAP整合各种日志类型.提供了日志收集、日志查询、监控告警等相关功能。提供了graylog sidecar通过sidecar模式可以很方便的收集目标主机、容器的各种日志信息，无缝整合filebeat。搜索语法跟kibana类似,自带简单的监控告警功能提供了webhook email等方式。

部署

官方提供了docker docker-compose 手动部署等模式可供用户自行选择
本次采用docker部署

graylog依赖了ES和MongoDB 需要提前将ES 和 MongoDB部署完成并且保持与graylog之间网络互通

生成**password_secret** 和 root_password_sha2

#password 需要至少64位的随机数即可 如果有多个节点graylog需要保证password_secret是一致的
pwgen -N 1 -s 96
# root_password_sha2 作为WEB-UI 的管理员密码 默认用户名admin
echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

采用docker部署

docker run
-p 9000:9000 -p 12201:12201 -p 1514:1514 -p 5555:5555  -p 5044:5044
-e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/"
-e GRAYLOG_IS_MASTER="true"
-e GRAYLOG_PASSWORD_SECRET="填入上面生成的password_secret"
-e GRAYLOG_ROOT_PASSWORD_SHA2="填入上面生成的root_password"
-e GRAYLOG_ELASTICSEARCH_HOSTS="http://username:[email protected]:9200"
-e GRAYLOG_MONGODB_URI="mongodb://10.60.36.230:27017/graylog"
-e GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai
--name=graylog
-d graylog/graylog:5.0

访问graylog WEB-UI localhost:9000 默认端口9000 输入用户名admin和上面设置的root_password密码的原文

此上graylog-server部署完成

graylog sidecar

sidecar类似一个agent的模式作为日志的采集者发送数据给graylog-server也是官方比较推荐的一种日志收集方式

选择符合graylog-server版本的 sidecar https://go2docs.graylog.org/5-0/getting_in_log_data/graylog_sidecar.html?tocpath=Getting%20in%20Log%20Data%7CGraylog%20Sidecar%7C_____0
以windows为例其他安装参考官方文档
通过WEB-UI 生成一个API token
根据步骤逐步安装修改graylog API的URL 和刚刚生成API token 还有实例名称(默认用hostname)
如果上述配置错了可以再 C:\\Program Files\\Graylog\\sidecar\\sidecar.yml. 修改其中的配置重启
此时可以在WEB-UI看见sidecar
添加INPUT BEAT数据源
配置sidecar的configuration抓取规则选择collector收集器为filebeat 下方输入filebeat的抓取配置规则

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["localhost:5044"]
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
 - windows
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\Users\admin\Desktop\test_log\info/*.log
  multiline.pattern: '^20'                    #多行匹配规则
  multiline.negate: true                    #将不匹配的规则的行合并在一起
  multiline.match: after                #合并到匹配规则的上一行末尾
  tags: ["demo","info"]

- type: log
  enabled: true
  paths:
    - C:\Users\admin\Desktop\test_log\*-error-*.log
  multiline.pattern: '^20'                    #多行匹配规则
  multiline.negate: true                    #将不匹配的规则的行合并在一起
  multiline.match: after                #合并到匹配规则的上一行末尾
  tags: ["demo","error"]

下发filebeat的配置文件 Assign Configuration 并且执行 Process
选择sidecar Show messages 查看日志

配置graylog 日志告警

选择顶部导航栏Alerts 选择 Notification 添加通知渠道
选择添加一个web hook的通知渠道
选择顶部导航栏Alerts 选择Event Definition 创建一个事件定义
根据提示步骤定义事件基础信息描述级别

配置filter 核心需要配置Search Query 规则右侧可以返回Search Query返回的数据
配置执行调度时间等规则
自定义字段如有需要的话
选择之前配置的通知渠道选择message backlog 会最终通知到webhook的backlog字段中包含了消息原文信息建议配置 grace period 配置安静期避免告警风暴
查看配置概述没问题选择Update event definition

发送ERROR日志查看webhook接口的告警

返回json报文示例

{
    "event_definition_id": "643775ac247b2e0934262df5",
    "event_definition_type": "aggregation-v1",
    "event_definition_title": "demo-error-log-alert",
    "event_definition_description": "",
    "job_definition_id": "6437759a247b2e0934262dcd",
    "job_trigger_id": "6437bfc8247b2e093426c914",
    "event": {
        "id": "01GXWWA9QSVY0G89SMD94YB0W2",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "643775ac247b2e0934262df5",
        "origin_context": "urn:graylog:message:es:graylog_0:a5250950-d9d6-11ed-bcc0-0242ac110002",
        "timestamp": "2023-04-13T08:38:54.824Z",
        "timestamp_processing": "2023-04-13T08:39:36.441Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
            
        ],
        "source_streams": [
            "000000000000000000000001"
        ],
        "message": "demo-error-log-alert",
        "source": "1ac9b7a6d52e",
        "key_tuple": [
            
        ],
        "key": "",
        "priority": 3,
        "alert": true,
        "fields": {
            
        },
        "group_by_fields": {
            
        }
    },
    "backlog": [
        {
            "index": "graylog_0",
            "message": "2023-04-13 16:38:54.824 -- [demo1] -- ERROR 14996 --- [http-nio-4100-exec-5] [] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause\n\njava.lang.ArithmeticException: / by zero\n\tat com.corn.controller.LogController.error(LogController.java:26)\n\tat sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:655)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:748)",
            "fields": {
                "filebeat_log_offset": 186267,
                "filebeat_agent_name": "LAPTOP-1BQNQ8EO",
                "gl2_remote_ip": "172.17.0.1",
                "gl2_remote_port": 59776,
                "filebeat_agent_hostname": "LAPTOP-1BQNQ8EO",
                "beats_type": "filebeat",
                "gl2_source_input": "64377490247b2e0934262b8e",
                "filebeat_@metadata_beat": "filebeat",
                "filebeat_@timestamp": "2023-04-13T08:38:54.824Z",
                "filebeat_agent_type": "filebeat",
                "filebeat_@metadata_version": "7.11.1",
                "filebeat_host_name": "LAPTOP-1BQNQ8EO",
                "gl2_source_node": "95acd4bd-4f23-40de-a422-5274f220a40a",
                "filebeat_agent_version": "7.11.1",
                "filebeat_agent_ephemeral_id": "d5392ef5-44f5-4d83-9007-1c947788c0f2",
                "gl2_accounted_message_size": 5774,
                "gl2_source_collector": "41e84e5f-67c5-422f-8fb0-a5205146cd4b",
                "filebeat_input_type": "log",
                "gl2_message_id": "01GXWW99K54AZYA4JCYQ0CT0ZG",
                "filebeat_tags": [
                    "windows",
                    "demo",
                    "error"
                ],
                "filebeat_ecs_version": "1.6.0",
                "filebeat_collector_node_id": "LAPTOP-1BQNQ8EO",
                "filebeat_@metadata_type": "_doc",
                "filebeat_log_time": "2023-04-13 16:38:54.824",
                "filebeat_agent_id": "4b8086ce-67f1-4aec-b93b-723744d56ab3",
                "filebeat_log_file_path": "C:\\Users\\JimWu\\Desktop\\test_log\\demo1-error-2023-04-13-0.log",
                "filebeat_log_flags": [
                    "multiline"
                ]
            },
            "id": "a5250950-d9d6-11ed-bcc0-0242ac110002",
            "source": "LAPTOP-1BQNQ8EO",
            "timestamp": "2023-04-13T08:38:54.824Z",
            "stream_ids": [
                "000000000000000000000001"
            ]
        }
    ]
}

graylog实现日志监控

graylog

部署

graylog sidecar

配置graylog 日志告警

悦读