Bootstrap

FileBeat部署及基础使用

一、FileBeat部署及基础使用

1.部署FileBeat环境

yum -y localinstall filebeat-7.17.6-x86_64.rpm

2.修改FileBeat的配置⽂件

(1)编写测试的配置⽂件
mkdir /etc/filebeat/config
cat > /etc/filebeat/config/01-stdin-to-console.yml <<'EOF'
# 指定输⼊的类型
filebeat.inputs:
# 指定输⼊的类型为"stdin",表示标准输⼊
- type: stdin
# 指定输出的类型
output.console:
  # 打印漂亮的格式
  pretty: true
EOF

(2)运⾏filebeat实例
filebeat -e -c /etc/filebeat/config/01-stdin-to-console.yml 

(3)测试
⻅视频。

3.FileBeat的Log类型

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log

output.console:
  pretty: true

4.input的通配符案例

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log
    - /tmp/*.txt

output.console:
  pretty: true

5.input的通用字段案例

filebeat.inputs:
- type: log
  # 是否启动当前的输⼊类型，默认值为true
  enabled: true
  # 指定数据路径
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  # 给当前的输⼊类型搭上标签
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
  # ⾃定义字段
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  # 将⾃定义字段的key-value放到顶级字段.
  # 默认值为false，会将数据放在⼀个叫"fields"字段的下⾯.
  fields_under_root: true

output.console:
  pretty: true

6.日志过滤案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test/*.log
  # 注意，⿊⽩名单均⽀持通配符,⽣产环节中不建议同时使⽤
  # 指定⽩名单，包含指定的内容才会采集，且区分⼤⼩写!
  include_lines: ['^ERR', '^WARN','oldboyedu']
  # 指定⿊名单，排除指定的内容
  exclude_lines: ['^DBG',"linux","oldboyedu"]

output.console:
  pretty: true

7.将数据写入ES案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]

8.自定义es索引名称

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"

9.多个索引写入案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  # index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
      # 匹配指定字段包含的内容
      when.contains:
        tags: "oldboyedu-linux80"
    - index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"

10.自定义分片和副本案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  # index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
      # 匹配指定字段包含的内容
      when.contains:
        tags: "oldboyedu-linux80"
    - index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"
# 覆盖已有的索引模板
setup.template.overwrite: false
# 配置索引模板
setup.template.settings:
  # 设置分⽚数量
  index.number_of_shards: 3
  # 设置副本数量，要求⼩于集群的数量
  index.number_of_replicas: 2

11.FileBeat实现日志聚合到本地

filebeat.inputs:
- type: tcp
  host: "0.0.0.0:9000"

output.file:
  path: "/tmp/filebeat"
  filename: oldboyedu-linux80
  # 指定⽂件的滚动⼤⼩，默认值为20MB
  rotate_every_kb: 102400
  # 指定保存的⽂件个数，默认是7个，有效值为2-1024个
  number_of_files: 300
  # 指定⽂件的权限，默认权限是0600
  permissions: 0600

12.FileBeat实现日志聚合到ES集群

在这里插入图片描述

filebeat.inputs:
- type: tcp
  host: "0.0.0.0:9000"
  tags: ["aaa"]

- type: tcp
  host: "0.0.0.0:8000"
  tags: ["bbb"]

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
  indices:
    - index: "oldboyedu-linux80-elk-aaa-%{+yyyy.MM.dd}"
      when.contains:
        tags: "aaa"

    - index: "oldboyedu-linux80-elk-bbb-%{+yyyy.MM.dd}"
      when.contains:
        tags: "bbb"

setup.ilm.enabled: false
setup.template.name: "oldboyedu-linux80-elk"
setup.template.pattern: "oldboyedu-linux80-elk*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0

13.FileBeat的Log类型删除不必要的字段

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log

output.console:
  pretty: true

processors:
  - drop_fields:
      fields: ['agent','ecs', 'beat', 'input_type', 'tags', 'count', '@version', 'log', 'offset', 'type', 'host','@metadata','input']

悦读

道可道，非常道；名可名，非常名。 无名，天地之始，有名，万物之母。 故常无欲，以观其妙，常有欲，以观其徼。 此两者，同出而异名，同谓之玄，玄之又玄，众妙之门。

最新收录
新手小白的pytorch学习第十三弹------卷积神经网络CNN和保存加载模型参数save and load model state dict
基于SpringBoot+Vue点餐系统设计和实现(源码+LW+部署讲解)
SpringBoot漏洞
【OpenJudge NOI】题解目录
C语言输出教学日历表
WSDM2024推荐系统和LLM相关论文整理（一）
aws(学习笔记第十四课) 面向NoSQL DB的DynamoDB
前端Canvas组件学习
[Unity] 创建网格顶点和三角形
多线程详解
;