# 一、在Filebeat写好配置文件

```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log*
  tags: ["access"]
  fields:
    name: "access"

- type: log
  enabled: true
  paths:
    -  /usr/local/nginx/logs/error.log*
  tags: ["error"]
  fields:
    name: "error"
    
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "nginx-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "nginx-elk-access-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"
    - index: "nginx-elk-error-%{+yyyy.MM.dd}"
      when.contains:
        tags: "error"

# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引模板
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"
#设置已有索引模版直接覆盖
setup.template.overwrite: true

#设置索引模版分片
setup.template.settings:
    #设置分片数
    index.number_of_shards: 3
    #设置副本数
    index.number_of_replicas: 0
```
在nginx服务器上启动Filebeat
```js
filebeat -e -c /etc/filebeat/filebeat.yml
```

![image.png](/static/img/610238ba4ef2d5e3ce5e65988ccc39be.image.webp)

然后去索引管理创建好索引模式
![image.png](/static/img/11a99c9e86e975cd9f8ba2dcc2a2d09e.image.webp)
最后查看已经完成配置
![image.png](/static/img/432876178efecccc020a5e1d67a82f15.image.webp)


# 二、这种方式查看，不会把字段分出来

![image.png](/static/img/17c900146c150991c9b791fc9991971e.image.webp)

修改nginx.conf
```js
worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    include   songxuan.conf; #配置外置文件

    # 定义JSON格式日志
    log_format json_combined escape=json
    '{'
        '"timestamp":"$time_iso8601",'
        '"remote_addr":"$remote_addr",'
        '"remote_user":"$remote_user",'
        '"request":"$request",'
        '"status": "$status",'
        '"body_bytes_sent":"$body_bytes_sent",'
        '"request_time":"$request_time",'
        '"http_referer":"$http_referer",'
        '"http_user_agent":"$http_user_agent",'
        '"http_x_forwarded_for":"$http_x_forwarded_for"'
    '}';
    # 指定访问日志路径和格式
    access_log /usr/local/nginx/logs/access.log json_combined;
    error_log  /usr/local/nginx/logs/error.log;

}

```
```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log*
  tags: ["access"]
  
  #添加json格式
  json.keys_under_root: true
  json.add_error_key: true
  
  fields:
    name: "access"

- type: log
  enabled: true
  paths:
    -  /usr/local/nginx/logs/error.log*
  tags: ["error"]
  fields:
    name: "error"
    
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "nginx-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "nginx-elk-access-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"
    - index: "nginx-elk-error-%{+yyyy.MM.dd}"
      when.contains:
        tags: "error"

# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引模板
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"
#设置已有索引模版直接覆盖
setup.template.overwrite: true

#设置索引模版分片
setup.template.settings:
    #设置分片数
    index.number_of_shards: 3
    #设置副本数
    index.number_of_replicas: 0
```

这样配置查看能，能区别更清楚字段

![image.png](/static/img/7cccab639f232d2fc4860573d2934156.image.webp)


4.ES+kibana+Filebeat收集Nginx日志

# 一、配置自定义索引

```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维工程师"]
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原生开发"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "mysongxuan-elk-%{+yyyy.MM.dd}"
  
# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引名字
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"

```

/var/lib/filebeat/ 存储Filebeat的注册表文件（registry），记录日志文件的采集状态（如offset、inode等）
删除该目录会‌重置采集状态‌，但不会直接影响模板加载
```js
rm -rf /var/lib/filebeat/*
```

重新配置文件后启动，我的自定义索引模版就出来了

![image.png](/static/img/8ad91fe68e640ce4058c3f4457bde1f9.image.webp)
然后创建索引模式，就可以看到我自定义的索引模式了
![image.png](/static/img/a88a6b65f559c435363fc3b48bae9410.image.webp)

![image.png](/static/img/3415397d784cd47e894f0ee3ed9a9f54.image.webp)






# 二、根据不同的tags进行索引查询

```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/*.txt
  tags: ["songxuan","xuan"]
  fields:
    name: "松轩"
    class: "linux"

- type: log
  enabled: true
  paths:
    - /tmp/*.log
  tags: ["xiaoming","xiaoqiang"]
  fields:
    name: "小明"
    hobby: "python"
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "mysongxuan-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "mysongxuan-elk-%{+yyyy.MM.dd}"
      when.contains:
        tags: "songxuan"
    - index: "mysongxuan-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "xiaoming"

# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引模板
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"
```
删除后启动
```js
rm -rf /var/lib/filebeat/*
```

```js
filebeat -e -c /etc/filebeat/filebeat.yml
```
重新查看后有就出现两条索引了，
![image.png](/static/img/8d7d3548745e6656089372121608136b.image.webp)

最后去索引模式创建索引

![image.png](/static/img/4c0249b73fb3dc5646b0713b9d6b91c5.image.webp)

然后查看标签是songxuan的只有对应文本的数据

![image.png](/static/img/44624a62306a5147602c98578dcdbd2e.image.webp)

![image.png](/static/img/1ed457164c2579b9b81d34a7d615ffaa.image.webp)



3.自定义索引

#  一、log日志配置

```js
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /tmp/test.log

output.console:
  pretty: true
```
```js
echo 123 > test.log
```

![image.png](/static/img/7fa4344ce5bdc86b1e21cb14bd24b952.image.webp)


这是他的数据原文件目录，中断会跟局该目录下的log.json进行读取。
```js
cd /var/lib/filebeat/registry/filebeat/
```

也可以用通配符去匹配

```js
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /tmp/test.log
      - /tmp/*.txt

output.console:
  pretty: true
```


加标签
```yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维工程师"]
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原生开发"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.console:
  pretty: true

```

运行命令
```js
filebeat -e -c /etc/filebeat/filebeat.yml
```



# 二、把收集的数据发送到集群上
写上Elasticsearch集群的地址配置上去。
```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维工程师"]
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原生开发"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
```

启动后去Kibana页面查看
![image.png](/static/img/e70a427525165e9743f802a156cb4c5c.image.webp)

![image.png](/static/img/0cb1401e7cdfcb92e85b054cfc542e7f.image.webp)


# 三、创建索引模式

![image.png](/static/img/f1524b861db3b10fc4cedb79db41174b.image.webp)

可以通过别名＋通配符的方式去匹配

![image.png](/static/img/d2b8acac3364e172b1d7b60ae195afff.image.webp)

然后创建
![image.png](/static/img/faaf8b6c441a43d965c58636828ba407.image.webp)


最后点击这里可以查看采集到日志的数据

![image.png](/static/img/eceba1bf91ee8a64bad6a5021e2f6fbf.image.webp)


# 四、筛选字段查看

![image.png](/static/img/c2547960429ae363ce79a91cd5f1de37.image.webp)

当我去给文本添加内容时，也是能收集到的
![image.png](/static/img/8b09332bc4d041e2f20c70935750e362.image.webp)


![image.png](/static/img/aa4a188c7ece2dd59f7ced60e2eda371.image.webp)

2.Filebeat配置

# 一、环境配置
1.准备了三台机器
```js
192.168.201.112
192.168.201.113
192.168.201.114
```

```js
hostnamectl set-hostname  ELK01
hostnamectl set-hostname  ELK02
hostnamectl set-hostname  ELK03
```


```js
cat >> /etc/hosts <<EOF
192.168.201.112 ELK01
192.168.201.113  ELK02
192.168.201.114  ELK03
EOF
```
2.临时关闭防火墙

```js
systemctl stop firewalld
```
3.永久关闭防火墙

```js
systemctl disable firewalld
```


4.关闭内存交换分区

```js
swapoff -a
```

```js
vim /etc/fstab
# 注释 swap 行
```

5.关闭selinux

```js
setenforce 0	
```
6.同步服务器时间

```js
yum install ntpdate -t
ntpdate -u ntp.aliyun.com
timedatectl set-timezone Asia/Shanghai
```

7.配置免密

A服务器 配置免密登录 如果没有~/.ssh目录下没有东西或没目录，用以下命令重新生成一下
```js
# 强制创建.ssh目录并设置权限
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# 生成RSA密钥对（无密码）
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""

# 设置密钥权限
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub

# 将公钥追加到authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```
把 A服务器 的公钥给到 B服务器，把 A服务器 的公钥放到 B服务器 的~/.ssh/authorized_keys里面

```js
#A服务器执行
cat id_rsa.pub
```

```js
#B服务器执行，然后把A服务器执行的查看到的公钥复制到B服务器
vi ~/.ssh/authorized_keys
```

# 二、安装Elasticsearch 7.17.29(单机部署)
核心功能：分布式搜索和分析引擎，负责存储、索引和快速检索数据（如日志、指标等）。

特点：支持实时搜索、高扩展性，通过倒排索引实现高效查询。


下载地址：https://www.elastic.co/downloads/past-releases/elasticsearch-7-17-29

rpm部署
```js
rpm elasticsearch-7.17.29-x86_64.rpm
```
启动服务
```js
systemctl start elasticsearch.service
```
修改配置文件
```js
vi /etc/elasticsearch/elasticsearch.yml
```

```yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: my-elk
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: elk01
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.201.112","192.168.201.113","192.168.201.114"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don't have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features. 
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html
```

![image.png](/static/img/6f0d355ba19e2b4f97b9f194b5634ed5.image.webp)


# 三、集群部署
都修改yml文件
```js
vi /etc/elasticsearch/elasticsearch.yml
```


```js
#改成对应的elk，如192.168.201.112是elk01,192.168.201.112是elk02
node.name: elk01

discovery.seed_hosts: ["192.168.201.112","192.168.201.113","192.168.201.114"]
```

测试集群是否都连上了
```js
[root@elk01 elasticsearch]# curl 192.168.201.113:9200/_cat/nodes?v
ip              heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
192.168.201.114           11          97   0    0.03    0.04     0.05 cdfhilmrstw -      elk03
192.168.201.112           25          96   0    0.00    0.01     0.05 cdfhilmrstw *      elk01
192.168.201.113           10          96   0    0.00    0.01     0.05 cdfhilmrstw -      elk02
```

# 四、Kibana安装
核心功能：数据可视化平台，提供图表、仪表盘等工具，用于展示和分析Elasticsearch中的数据。

特点：支持交互式查询和自定义视图，便于监控和故障排查。

https://www.elastic.co/downloads/past-releases/kibana-7-17-29


```js
rpm -ivh kibana-7.17.29-x86_64.rpm
```
修改配置文件
```js
vi /etc/kibana/kibana.yml
```

```js

server.host: "0.0.0.0"

server.name: "elk"


elasticsearch.hosts: ["http://192.168.201.112:9200","http://192.168.201.113:9200","http://192.168.201.114:9200"]


i18n.locale: "zh-CN"
```

```js
systemctl start  kibana
```

![image.png](/static/img/288fc4bee7b1b31fad752717ede1ff1a.image.webp)




# 五、Filebeat
轻量级：相比Logstash，Filebeat占用资源极少，适合部署在边缘节点（如服务器、容器）持续采集日志文件。

专一性：仅负责日志收集和转发（到Logstash或Elasticsearch），不处理数据，避免资源浪费。

场景互补：Filebeat + Logstash组合中，Filebeat负责采集，Logstash负责复杂处理，分工明确提升效率。

下载命令
```js
wget  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.29-x86_64.rpm
```

```js
vi /etc/filebeat/filebeat.yml 
```
编写一个测试数据
```js
filebeat.inputs:
  - type: stdin

output.console:
  pretty: true
```
指定配置文件启动
```js
filebeat -e -c /etc/filebeat/filebeat.yml
```
![image.png](/static/img/5b3e8defcb4963da720da1bf0e9b1711.image.webp)


1.ELK环境配置和安装

Linux免密登录确实是指A服务器通过SSH协议登录B服务器时无需手动输入密码。其核心原理是通过非对称加密的密钥对实现身份验证：A服务器生成公钥和私钥后，将公钥部署到B服务器的授权文件中，后续登录时B服务器会用该公钥验证A服务器的私钥，从而跳过密码验证环节

# A服务器 配置免密登录 如果没有~/.ssh目录下没有东西或没目录，用以下命令重新生成一下

```js
# 强制创建.ssh目录并设置权限
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# 生成RSA密钥对（无密码）
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""

# 设置密钥权限
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub

# 将公钥追加到authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```
# 把 A服务器 的公钥给到 B服务器

```js
$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGYg72GXNQmFPwEEp986OqHF7vCcVxJw6mox5iKifJReaUKdcLEXqjUcu6mUi29j3hFMi4QT7Oc3+JmgEeAxWLWmC4zADm0NWqzKDJ6qdptU01cMkFlukpBU9u/GBAyCLQ6sgK75O62HTIq2MVe9TqXOficr90IOSxpObPFq1ZEnWp6i76Ag7hMO3e8Hm4wSD4e6edxbDIgzzRNo58GO1YikHEiYkd86sMzJTI/Xf2myw9ETgen7MlFGrOgdBPktmD4Wexg3Ug72uZJxOB96sn5TOXzIYs9YsGvJeOsZ/zTk9okrRCIybjBjDJvmfDmjUNFF77byIFHCu2UAWpYGHeui3BlVwSjRdaysHIPFUEmadUwi0Oa3JpgnhU600UfsEJn3fGVm2XrwtLR0bhrj/LTsvM0UmvsAs0RC4b8EzrExbPWPfQPwUJLxKZcEfyW1cK6a4y8+qb2WV7HXv7acBtywTwBHAuuYeo6HKweyYoz/c2/pTWjZ3PpGETBUYB6km3WWrjDEzfxYVlGFb5mB+GWh1m5Knak9wABv06k9XVt3NL6ZjzkwH0WBd5Lr0YvfFocLLurq8gGjaBiWQ0lOPjDqILLwaV6Xm9ZoXyUb2My9yIBwYm2w9VhA9LreI0H8G8JDOzF+hBYV8ikF/8meqFuJgJMGe+/RZSNDhU/3FKQw== jenkins@5ca614ac051b
```
# 把 A服务器 的公钥放到 B服务器 的~/.ssh/authorized_keys里面

```js
vi ~/.ssh/authorized_keys
```

![b82dcae1b8802f9c7354499e1702fdaa_7cefdafdd8eedaf7e9279a361a62bcf4.image.webp](/static/img/11a4bd43ecf5857745bf55fc8a382798.b82dcae1b8802f9c7354499e1702fdaa_7cefdafdd8eedaf7e9279a361a62bcf4.image.webp)

# 然后测试免密是否成功，在A服务器执行ssh命令

![ef3cec66aa33ccbc67267156a03761bb_2ba284f1250e5c5d368c86ed2841b4ee.image.webp](/static/img/e1930cb3aa1c08df48a22f6e606b7df6.ef3cec66aa33ccbc67267156a03761bb_2ba284f1250e5c5d368c86ed2841b4ee.image.webp)