# 一、介绍和安装


```js
核心功能：数据收集、转换和传输管道。支持从多种来源（文件、数据库等）提取数据，过滤（如解析JSON、正则匹
配）后发送到Elasticsearch。

特点：功能强大但资源消耗较高，适合复杂的数据处理场景。

相比Logstash，Filebeat占用资源极少，适合部署在边缘节点（如服务器、容器）持续采集日志文件。

专一性：仅负责日志收集和转发（到Logstash或Elasticsearch），不处理数据，避免资源浪费。

场景互补：Filebeat + Logstash组合中，Filebeat负责采集，Logstash负责复杂处理，分工明确提升效率。
```
下载网址：https://artifacts.elastic.co/downloads/logstash/logstash-7.17.29-x86_64.rpm

```js
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.29-x86_64.rpm
```

```js
rpm -ivh logstash-7.17.29-x86_64.rpm
```

```js
ln  -sv /usr/share/logstash/bin/logstash  /usr/local/bin/
```
# 二、收集TCP端口日志
配置目录
```js
/etc/logstash
```

```js
vi /etc/logstash/logstash-sample.conf
```
```js
input {
  file {
    path => ["/tmp/*.txt"]  # 监听 /tmp 目录下的 .txt 文件
    start_position => "beginning"  # 从文件开头读取
  }
}

output {
  stdout {}  # 输出到控制台（测试用）
}
```
知道配置启动
```js
logstash -f /etc/logstash/logstash-sample.conf
```

![image.png](/static/img/7832b32a36879835035d7bbd7b380320.image.webp)

# 三、收集TCP端口日志

```js
input {
  tcp {
    port => 8888
    codec => "json"  # 可选：如果接收JSON格式数据
  }
}

output {
  stdout {
    codec => rubydebug  # 更好的格式化输出
  }
}
```
启动后去另一台服务器telnet该服务器端口测试

![image.png](/static/img/52909a85a3cfb1d68491594572abffce.image.webp)

成功收集日志
![image.png](/static/img/de6912db1d0f4a52d97dcd11374a70e2.image.webp)

# 三、收集http协议端口日志

```js
input {
  http {
    port => 8888
  }
}

output {
  stdout {
    codec => rubydebug  # 更好的格式化输出
  }
}
```
测试

```js
curl "http://192.168.201.104:8888?greeting=你好啊"
```

![image.png](/static/img/8edd571194859b7120eba60d2548379b.image.webp)

# 三、发送数据输出到redis环境

```js

input {
  tcp {
    port => 9999
  }
}

output {
  stdout {}
  
  redis {
    host => "192.168.201.104"
    port => 6379
    db => 0
    password => "12345678"
    data_type => "list"
    key => "hello word"
  }
}

```
我往该IP端口发送消息，就会往redis写入数据
![image.png](/static/img/848854c44de8221511db0c4112b02912.image.webp)


![image.png](/static/img/1dd28b1b703b4402fde651cd1fc620b6.image.webp)

# 四、发送数据输出到本地文件

```js
input {
  tcp {
    port => 9999
  }
}

output {
  stdout {}
  
 file {
    path => ["/tmp/1.txt"]  # 监听 /tmp 目录下的 .txt 文件
  }
}
```
发送消息写入该服务器下的路径文件
![image.png](/static/img/5b2138ab4135fa3067d18950466a900f.image.webp)

![image.png](/static/img/71299cac68c2794994dcd442267949ba.image.webp)


# 五、发送数据输出到ES

```js
input {
  tcp {
    port => 9999
  }
}

output {
  stdout {}
  

  elasticsearch {
    hosts => ["192.168.201.112:9200", "192.168.201.113:9200", "192.168.201.114:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

```


![image.png](/static/img/a874ae8e7ee48a1faf5ea6133f303bc7.image.webp)


![image.png](/static/img/73b9b05ddfe640293ae7e70ac0704a43.image.webp)

5.logstash

# 一、在Filebeat写好配置文件

```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log*
  tags: ["access"]
  fields:
    name: "access"

- type: log
  enabled: true
  paths:
    -  /usr/local/nginx/logs/error.log*
  tags: ["error"]
  fields:
    name: "error"
    
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "nginx-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "nginx-elk-access-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"
    - index: "nginx-elk-error-%{+yyyy.MM.dd}"
      when.contains:
        tags: "error"

# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引模板
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"
#设置已有索引模版直接覆盖
setup.template.overwrite: true

#设置索引模版分片
setup.template.settings:
    #设置分片数
    index.number_of_shards: 3
    #设置副本数
    index.number_of_replicas: 0
```
在nginx服务器上启动Filebeat
```js
filebeat -e -c /etc/filebeat/filebeat.yml
```

![image.png](/static/img/610238ba4ef2d5e3ce5e65988ccc39be.image.webp)

然后去索引管理创建好索引模式
![image.png](/static/img/11a99c9e86e975cd9f8ba2dcc2a2d09e.image.webp)
最后查看已经完成配置
![image.png](/static/img/432876178efecccc020a5e1d67a82f15.image.webp)


# 二、这种方式查看，不会把字段分出来

![image.png](/static/img/17c900146c150991c9b791fc9991971e.image.webp)

修改nginx.conf
```js
worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    include   songxuan.conf; #配置外置文件

    # 定义JSON格式日志
    log_format json_combined escape=json
    '{'
        '"timestamp":"$time_iso8601",'
        '"remote_addr":"$remote_addr",'
        '"remote_user":"$remote_user",'
        '"request":"$request",'
        '"status": "$status",'
        '"body_bytes_sent":"$body_bytes_sent",'
        '"request_time":"$request_time",'
        '"http_referer":"$http_referer",'
        '"http_user_agent":"$http_user_agent",'
        '"http_x_forwarded_for":"$http_x_forwarded_for"'
    '}';
    # 指定访问日志路径和格式
    access_log /usr/local/nginx/logs/access.log json_combined;
    error_log  /usr/local/nginx/logs/error.log;

}

```
```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log*
  tags: ["access"]
  
  #添加json格式
  json.keys_under_root: true
  json.add_error_key: true
  
  fields:
    name: "access"

- type: log
  enabled: true
  paths:
    -  /usr/local/nginx/logs/error.log*
  tags: ["error"]
  fields:
    name: "error"
    
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "nginx-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "nginx-elk-access-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"
    - index: "nginx-elk-error-%{+yyyy.MM.dd}"
      when.contains:
        tags: "error"

# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引模板
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"
#设置已有索引模版直接覆盖
setup.template.overwrite: true

#设置索引模版分片
setup.template.settings:
    #设置分片数
    index.number_of_shards: 3
    #设置副本数
    index.number_of_replicas: 0
```

这样配置查看能，能区别更清楚字段

![image.png](/static/img/7cccab639f232d2fc4860573d2934156.image.webp)


4.ES+kibana+Filebeat收集Nginx日志

# 一、配置自定义索引

```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维工程师"]
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原生开发"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "mysongxuan-elk-%{+yyyy.MM.dd}"
  
# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引名字
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"

```

/var/lib/filebeat/ 存储Filebeat的注册表文件（registry），记录日志文件的采集状态（如offset、inode等）
删除该目录会‌重置采集状态‌，但不会直接影响模板加载
```js
rm -rf /var/lib/filebeat/*
```

重新配置文件后启动，我的自定义索引模版就出来了

![image.png](/static/img/8ad91fe68e640ce4058c3f4457bde1f9.image.webp)
然后创建索引模式，就可以看到我自定义的索引模式了
![image.png](/static/img/a88a6b65f559c435363fc3b48bae9410.image.webp)

![image.png](/static/img/3415397d784cd47e894f0ee3ed9a9f54.image.webp)






# 二、根据不同的tags进行索引查询

```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/*.txt
  tags: ["songxuan","xuan"]
  fields:
    name: "松轩"
    class: "linux"

- type: log
  enabled: true
  paths:
    - /tmp/*.log
  tags: ["xiaoming","xiaoqiang"]
  fields:
    name: "小明"
    hobby: "python"
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
  index: "mysongxuan-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "mysongxuan-elk-%{+yyyy.MM.dd}"
      when.contains:
        tags: "songxuan"
    - index: "mysongxuan-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "xiaoming"

# 禁用索引生命周期管理
setup.ilm.enabled: false
# 设置索引模板
setup.template.name: "mysongxuan-elk"
setup.template.pattern: "mysongxuan-elk*"
```
删除后启动
```js
rm -rf /var/lib/filebeat/*
```

```js
filebeat -e -c /etc/filebeat/filebeat.yml
```
重新查看后有就出现两条索引了，
![image.png](/static/img/8d7d3548745e6656089372121608136b.image.webp)

最后去索引模式创建索引

![image.png](/static/img/4c0249b73fb3dc5646b0713b9d6b91c5.image.webp)

然后查看标签是songxuan的只有对应文本的数据

![image.png](/static/img/44624a62306a5147602c98578dcdbd2e.image.webp)

![image.png](/static/img/1ed457164c2579b9b81d34a7d615ffaa.image.webp)



3.自定义索引

#  一、log日志配置

```js
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /tmp/test.log

output.console:
  pretty: true
```
```js
echo 123 > test.log
```

![image.png](/static/img/7fa4344ce5bdc86b1e21cb14bd24b952.image.webp)


这是他的数据原文件目录，中断会跟局该目录下的log.json进行读取。
```js
cd /var/lib/filebeat/registry/filebeat/
```

也可以用通配符去匹配

```js
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /tmp/test.log
      - /tmp/*.txt

output.console:
  pretty: true
```


加标签
```yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维工程师"]
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原生开发"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.console:
  pretty: true

```

运行命令
```js
filebeat -e -c /etc/filebeat/filebeat.yml
```



# 二、把收集的数据发送到集群上
写上Elasticsearch集群的地址配置上去。
```yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维工程师"]
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原生开发"]
  fields:
    name: "oldboy"
    hobby: "linux,抖音"
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://192.168.201.112:9200", "http://192.168.201.113:9200", "http://192.168.201.114:9200"]
```

启动后去Kibana页面查看
![image.png](/static/img/e70a427525165e9743f802a156cb4c5c.image.webp)

![image.png](/static/img/0cb1401e7cdfcb92e85b054cfc542e7f.image.webp)


# 三、创建索引模式

![image.png](/static/img/f1524b861db3b10fc4cedb79db41174b.image.webp)

可以通过别名＋通配符的方式去匹配

![image.png](/static/img/d2b8acac3364e172b1d7b60ae195afff.image.webp)

然后创建
![image.png](/static/img/faaf8b6c441a43d965c58636828ba407.image.webp)


最后点击这里可以查看采集到日志的数据

![image.png](/static/img/eceba1bf91ee8a64bad6a5021e2f6fbf.image.webp)


# 四、筛选字段查看

![image.png](/static/img/c2547960429ae363ce79a91cd5f1de37.image.webp)

当我去给文本添加内容时，也是能收集到的
![image.png](/static/img/8b09332bc4d041e2f20c70935750e362.image.webp)


![image.png](/static/img/aa4a188c7ece2dd59f7ced60e2eda371.image.webp)

2.Filebeat配置

# 一、环境配置
1.准备了三台机器
```js
192.168.201.112
192.168.201.113
192.168.201.114
```

```js
hostnamectl set-hostname  ELK01
hostnamectl set-hostname  ELK02
hostnamectl set-hostname  ELK03
```


```js
cat >> /etc/hosts <<EOF
192.168.201.112 ELK01
192.168.201.113  ELK02
192.168.201.114  ELK03
EOF
```
2.临时关闭防火墙

```js
systemctl stop firewalld
```
3.永久关闭防火墙

```js
systemctl disable firewalld
```


4.关闭内存交换分区

```js
swapoff -a
```

```js
vim /etc/fstab
# 注释 swap 行
```

5.关闭selinux

```js
setenforce 0	
```
6.同步服务器时间

```js
yum install ntpdate -t
ntpdate -u ntp.aliyun.com
timedatectl set-timezone Asia/Shanghai
```

7.配置免密

A服务器 配置免密登录 如果没有~/.ssh目录下没有东西或没目录，用以下命令重新生成一下
```js
# 强制创建.ssh目录并设置权限
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# 生成RSA密钥对（无密码）
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""

# 设置密钥权限
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub

# 将公钥追加到authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```
把 A服务器 的公钥给到 B服务器，把 A服务器 的公钥放到 B服务器 的~/.ssh/authorized_keys里面

```js
#A服务器执行
cat id_rsa.pub
```

```js
#B服务器执行，然后把A服务器执行的查看到的公钥复制到B服务器
vi ~/.ssh/authorized_keys
```

# 二、安装Elasticsearch 7.17.29(单机部署)
核心功能：分布式搜索和分析引擎，负责存储、索引和快速检索数据（如日志、指标等）。

特点：支持实时搜索、高扩展性，通过倒排索引实现高效查询。


下载地址：https://www.elastic.co/downloads/past-releases/elasticsearch-7-17-29

rpm部署
```js
rpm elasticsearch-7.17.29-x86_64.rpm
```
启动服务
```js
systemctl start elasticsearch.service
```
修改配置文件
```js
vi /etc/elasticsearch/elasticsearch.yml
```

```yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: my-elk
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: elk01
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.201.112","192.168.201.113","192.168.201.114"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don't have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features. 
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html
```

![image.png](/static/img/6f0d355ba19e2b4f97b9f194b5634ed5.image.webp)


# 三、集群部署
都修改yml文件
```js
vi /etc/elasticsearch/elasticsearch.yml
```


```js
#改成对应的elk，如192.168.201.112是elk01,192.168.201.112是elk02
node.name: elk01

discovery.seed_hosts: ["192.168.201.112","192.168.201.113","192.168.201.114"]
```

测试集群是否都连上了
```js
[root@elk01 elasticsearch]# curl 192.168.201.113:9200/_cat/nodes?v
ip              heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
192.168.201.114           11          97   0    0.03    0.04     0.05 cdfhilmrstw -      elk03
192.168.201.112           25          96   0    0.00    0.01     0.05 cdfhilmrstw *      elk01
192.168.201.113           10          96   0    0.00    0.01     0.05 cdfhilmrstw -      elk02
```

# 四、Kibana安装
核心功能：数据可视化平台，提供图表、仪表盘等工具，用于展示和分析Elasticsearch中的数据。

特点：支持交互式查询和自定义视图，便于监控和故障排查。

https://www.elastic.co/downloads/past-releases/kibana-7-17-29


```js
rpm -ivh kibana-7.17.29-x86_64.rpm
```
修改配置文件
```js
vi /etc/kibana/kibana.yml
```

```js

server.host: "0.0.0.0"

server.name: "elk"


elasticsearch.hosts: ["http://192.168.201.112:9200","http://192.168.201.113:9200","http://192.168.201.114:9200"]


i18n.locale: "zh-CN"
```

```js
systemctl start  kibana
```

![image.png](/static/img/288fc4bee7b1b31fad752717ede1ff1a.image.webp)




# 五、Filebeat
轻量级：相比Logstash，Filebeat占用资源极少，适合部署在边缘节点（如服务器、容器）持续采集日志文件。

专一性：仅负责日志收集和转发（到Logstash或Elasticsearch），不处理数据，避免资源浪费。

场景互补：Filebeat + Logstash组合中，Filebeat负责采集，Logstash负责复杂处理，分工明确提升效率。

下载命令
```js
wget  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.29-x86_64.rpm
```

```js
vi /etc/filebeat/filebeat.yml 
```
编写一个测试数据
```js
filebeat.inputs:
  - type: stdin

output.console:
  pretty: true
```
指定配置文件启动
```js
filebeat -e -c /etc/filebeat/filebeat.yml
```
![image.png](/static/img/5b3e8defcb4963da720da1bf0e9b1711.image.webp)
