# 一、介绍与安装
Alertmanager主要负责对Prometheus产生的告警进行统一处理，因此在Alertmanager配置中一般会包含以下几个主要部分：

● 全局配置（global）：用于定义一些全局的公共参数，如全局的SMTP配置，Slack配置等内容；

● 模板（templates）：用于定义告警通知时的模板，如HTML模板，邮件模板等；

● 告警路由（route）：根据标签匹配，确定当前告警应该如何处理；

● 接收人（receivers）：接收人是一个抽象的概念，它可以是一个邮箱也可以是微信，Slack或者Webhook等，接收
人一般配合告警路由使用；

● 抑制规则（inhibit_rules）：合理设置抑制规则可以减少垃圾告警的产生


```js
tar -zxvf alertmanager-0.29.0-rc.1.linux-amd64.tar.gz
cd alertmanager-0.29.0-rc.1.linux-amd64
```
启动
```js
./alertmanager
```
# 二、配置普罗米修斯

![image.png](/static/img/0fd006fa367330129f78dbfb84489dd7.image.webp)

在配置中修改
```yml
# 全局配置
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# 告警配置
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['192.168.201.110:9093']

# 规则文件配置
rule_files:
    - '/server/prometheus/prometheus-3.5.0.linux-amd64/alert_rules/*.rules'

# 监控目标配置
scrape_configs:
  # 监控Prometheus自身
  - job_name: "prometheus-server"
    file_sd_configs:
      - files:
          - /server/prometheus/node_exporter.yml
        refresh_interval: 5s

  # 监控Nginx
  - job_name: nginx
    static_configs:
      - targets: ['192.168.201.104:9113']
        labels:
          name: "nginx"

  # 监控Kubernetes
  - job_name: K8S
    static_configs:
      - targets: ['192.168.201.100:31666']
        labels:
          name: "k8s"

  # 监控Redis
  - job_name: redis
    static_configs:
      - targets: ['192.168.201.104:9121']
        labels:
          name: "redis"

  # 监控Docker
  - job_name: docker
    static_configs:
      - targets: ['192.168.201.104:8080']
        labels:
          name: "docker"

  # 监控MySQL
  - job_name: mysql
    static_configs:
      - targets: ['192.168.201.104:9104']
        labels:
          name: "mysql"

  # 监控Blackbox Exporter自身
  - job_name: 'blackbox-exporter'
    static_configs:
      - targets: ['117.72.79.70:9115']
        labels:
          name: "blackbox-exporter"

  # 通过Blackbox Exporter监控外部网站
#http配置
  - job_name: "blackbox_http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://www.baidu.com
        - https://songxuan.vip
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 117.72.79.70:9115
        
  # 监控告警alertmanager9093端口
  - job_name: alertmanager
    static_configs:
      - targets: ['192.168.201.110:9093']
        labels:
          name: "alertmanager"        
```

然后在普罗米修斯目录下创建一个alert_rules
```js
mkdir -p /server/prometheus/prometheus-3.5.0.linux-amd64/alert_rules
```
然后创建一个普罗米修斯的告警
```js
vim /server/prometheus/prometheus-3.5.0.linux-amd64/alert_rules/nginx.rules
```

```js
groups:
- name: nginx_alerts
  rules:
  - alert: NginxServiceDown
    # 修改后的表达式：同时检测指标为0和指标不存在的情况
    expr: nginx_up == 0 OR absent(nginx_up)
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Nginx服务不可用 (实例 {{ $labels.instance }})"
      description: "Nginx 服务已停止运行或无法访问超过1分钟。"
```
最后普罗米修斯这里也要配置好

![image.png](/static/img/bcf13b65a77cddac43b76e32cbb562f6.image.webp)
然后重启普罗米修斯。

# 三、查看普罗米修斯

![image.png](/static/img/6dd19eb83c784fc2e06f169d7dc4ba8e.image.webp)


![image.png](/static/img/bb079e928e7cd5f727555a5de66a1b73.image.webp)

Prometheus告警

# 一、使用docker-compose安装

```js
mkdir /data/mysql -p
cd /data/mysql
```


```yml
version: '3.1'
services:
  db:
    image: mysql:8.0
    restart: always
    container_name: mysql
    environment:
      TZ: Asia/Shanghai
      LANG: en_US.UTF-8
      MYSQL_ROOT_PASSWORD: 123456
    command:
      --default-authentication-plugin=mysql_native_password
      --character-set-server=utf8mb4
      --collation-server=utf8mb4_general_ci
      --lower_case_table_names=1
      --performance_schema=1
      --sql-mode=""
      --skip-log-bin
    volumes:
      - /data/mysql/data:/var/lib/mysql
    ports:
      - 3306:3306
```

最后启动
```js
docker-compose up -d
```
# 二、监控配置

```js
docker exec -it mysql mysql -uroot -p
```

```js
CREATE USER 'exporter'@'%' IDENTIFIED BY 'password' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
FLUSH PRIVILEGES;
```
验证
```js
docker exec -it mysql mysql -uexporter -p
```

在创建一个yml，用于监控mysql
```js
mkdir /data/mysqld_exporter -p
cd /data/mysqld_exporter
```


```js
vi docker-compose.yml
```
```yml
version: '3.3'
services:
  mysqld-exporter:
    image: prom/mysqld-exporter
    container_name: mysqld-exporter
    restart: always
    command:
      - '--collect.info_schema.processlist'
      - '--collect.info_schema.innodb_metrics'
      - '--collect.info_schema.tablestats'
      - '--collect.info_schema.tables'
      - '--collect.info_schema.userstats'
      - '--collect.engine_innodb_status'
      - '--config.my-cnf=/my.cnf'
    volumes:
      - ./my.cnf:/my.cnf
    ports:
      - 9104:9104
```
创建my.cnf配置文件

```js
cat > my.cnf << "EOF"
[client]
host=192.168.11.62
user=exporter
password=password
port=3306
EOF
```

然后启动
```js
docker-compose up -d
```
# 三、配置普罗米修斯

```yml
# 全局配置
global:
  # 数据抓取间隔（默认1分钟）
  scrape_interval: 15s
  # 告警规则评估频率（建议与抓取间隔一致）
  evaluation_interval: 15s
  # 抓取超时时间（默认10秒）

# 告警配置
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # 启用Alertmanager时需配置实际地址
          # - alertmanager:9093

# 规则文件配置
rule_files:
  # 告警规则文件路径（支持通配符）
  # - 'alert_rules/*.rules'
  # 记录规则文件路径
  # - 'recording_rules/*.rules'

# 监控目标配置
#监控linux
scrape_configs:
  - job_name: "prometheus-server"
    file_sd_configs:
      - files:
          - /server/prometheus/node_exporter.yml
        refresh_interval: 5s
    # 自定义标签（用于维度筛选）

#监控nginx
  - job_name: nginx
    static_configs:
      - targets: ['192.168.201.104:9113']
        # 自定义标签（用于维度筛选）
        labels:
          name: "nginx"
      
#监控k8s
  - job_name: K8S
    static_configs:
      - targets: ['192.168.201.100:31666']
        # 自定义标签（用于维度筛选）
        labels:
          name: "k8s"

#监控redis
  - job_name: redis
    static_configs:
      - targets: ['192.168.201.104:9121']
        # 自定义标签（用于维度筛选）
        labels:
          name: "redis"

#监控docker
  - job_name: docker
    static_configs:
      - targets: ['192.168.201.104:8080']
        # 自定义标签（用于维度筛选）
        labels:
          name: "docker"

#监控mysql
  - job_name: mysql
    static_configs:
      - targets: ['192.168.201.104:9104']
        # 自定义标签（用于维度筛选）
        labels:
          name: "mysql"
```

![image.png](/static/img/9e08e97d7cec007dcdd7f105c3471f94.image.webp)


模版7362
![image.png](/static/img/9efd6d232b2dd2f104e31000c6bc1dcb.image.webp)

10.Prometheus监控mysql

# 一、安装CAdvisor
为了能够获取到Docker容器的运行状态，用户可以通过Docker的stats命令获取到当前主机上运行容器的统计信息，可以查看容器的CPU利用率、内存使用量、网络IO总量以及磁盘IO总量等信息。


写一个docker-compose.yml文件
```yml
version: '3.8'

services:
  cadvisor:
    image: mirrorgooglecontainers/cadvisor:latest  # Docker Hub 上的替代镜像，国内可访问
    container_name: cadvisor
    restart: always
    network_mode: host
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
```

修改普罗米修斯文件
```yml
# 全局配置
global:
  # 数据抓取间隔（默认1分钟）
  scrape_interval: 15s
  # 告警规则评估频率（建议与抓取间隔一致）
  evaluation_interval: 15s
  # 抓取超时时间（默认10秒）

# 告警配置
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # 启用Alertmanager时需配置实际地址
          # - alertmanager:9093

# 规则文件配置
rule_files:
  # 告警规则文件路径（支持通配符）
  # - 'alert_rules/*.rules'
  # 记录规则文件路径
  # - 'recording_rules/*.rules'

# 监控目标配置
#监控linux
scrape_configs:
  - job_name: "prometheus-server"
    file_sd_configs:
      - files:
          - /server/prometheus/node_exporter.yml
        refresh_interval: 5s


#监控nginx
  - job_name: nginx
    static_configs:
      - targets: ['192.168.201.104:9113']
        # 自定义标签（用于维度筛选）
        labels:
          name: "nginx"
      
#监控k8s
  - job_name: K8S
    static_configs:
      - targets: ['192.168.201.100:31666']
        # 自定义标签（用于维度筛选）
        labels:
          name: "k8s"

#监控redis
  - job_name: redis
    static_configs:
      - targets: ['192.168.201.104:9121']
        # 自定义标签（用于维度筛选）
        labels:
          name: "redis"
          
#监控docker
  - job_name: docker
    static_configs:
      - targets: ['192.168.201.104:8080']
        # 自定义标签（用于维度筛选）
        labels:
          name: "docker"
```

![image.png](/static/img/7f232d47a476d900a35e8b6cdbfbb5c3.image.webp)


![image.png](/static/img/4ddabc609b5ef5ff191b2ba27e7de833.image.webp)

# 二、导入模板
9621

![image.png](/static/img/bf10eeda9beeed2007b63c2742465ae7.image.webp)

9.Prometheus监控Docker

# 一、安装和配置redis_exporter
redis_exporter下载地址：https://github.com/oliver006/redis_exporter/releases


```js
# 创建专用用户
sudo useradd -rs /bin/false redis_exporter


# 解压并安装
tar xvf redis_exporter-v1.79.0.linux-amd64.tar.gz
mv redis_exporter-v1.79.0.linux-amd64.tar.gz redis_exporter-v1.79.0 
mv redis_exporter-v1.79.0  /usr/local/bin/
```

创建Systemd服务文件

```js
vi /etc/systemd/system/redis_exporter.service
```
```js
[Unit]
Description=Redis Exporter
Documentation=https://github.com/oliver006/redis_exporter
After=network.target

[Service]
Type=simple
User=redis_exporter
Group=redis_exporter
ExecStart=/usr/local/bin/redis_exporter-v1.79.0/redis_exporter \
  --redis.addr=redis://192.168.201.104:6379 \
  --redis.password=12345678
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
```

```js
sudo systemctl daemon-reload
sudo systemctl enable --now redis_exporter
sudo systemctl start redis_exporter   #启动
sudo systemctl status redis_exporter  # 验证状态
```
# 二、修改普罗米修斯配置

```yml
# 全局配置
global:
  # 数据抓取间隔（默认1分钟）
  scrape_interval: 15s
  # 告警规则评估频率（建议与抓取间隔一致）
  evaluation_interval: 15s
  # 抓取超时时间（默认10秒）

# 告警配置
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # 启用Alertmanager时需配置实际地址
          # - alertmanager:9093

# 规则文件配置
rule_files:
  # 告警规则文件路径（支持通配符）
  # - 'alert_rules/*.rules'
  # 记录规则文件路径
  # - 'recording_rules/*.rules'

# 监控目标配置
#监控linux
scrape_configs:
  - job_name: "prometheus-server"
    file_sd_configs:
      - files:
          - /server/prometheus/node_exporter.yml
        refresh_interval: 5s


#监控nginx
  - job_name: nginx
    static_configs:
      - targets: ['192.168.201.104:9113']
        # 自定义标签（用于维度筛选）
        labels:
          name: "nginx"
      
#监控k8s
  - job_name: K8S
    static_configs:
      - targets: ['192.168.201.100:31666']
        # 自定义标签（用于维度筛选）
        labels:
          name: "k8s"

#监控redis
  - job_name: redis
    static_configs:
      - targets: ['192.168.201.104:9121']
        # 自定义标签（用于维度筛选）
        labels:
          name: "redis"

```


![image.png](/static/img/6e3ef11e1803fd5da457039f9aa45bc2.image.webp)

常用的监控指标
```js
redis_up                                    # 服务器是否在线
redis_uptime_in_seconds                     # 运行时长，单位 s
rate(redis_cpu_sys_seconds_total[1m]) + rate(redis_cpu_user_seconds_total[1m])  # 占用 CPU 核数
redis_memory_used_bytes                     # 占用内存量
redis_memory_max_bytes                      # 限制的最大内存，如果没限制则为 0
delta(redis_net_input_bytes_total[1m])      # 网络接收的 bytes
delta(redis_net_output_bytes_total[1m])     # 网络发送的 bytes


redis_connected_clients                     # 客户端连接数
redis_connected_clients / redis_config_maxclients # 连接数使用率
redis_rejected_connections_total            # 拒绝的客户端连接数
redis_connected_slaves                      # slave 连接数
```

导入模板，11835

![image.png](/static/img/649440d3bea0fe2181797cdbc00f385d.image.webp)

8.Prometheus监控Redis


```js

input {
  file {
    path => "/usr/local/nginx/logs/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "nginx_access"
  }
}

filter {
  grok {
    match => { 
      "message" => '%{IPORHOST:remote_addr} - %{DATA:remote_user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATH:request_path}(?:\?%{URIPARAM:query_params})? HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{DATA:http_referer}" "%{DATA:user_agent}"'
    }
  }

  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => [
      "192.168.201.112:9200",
      "192.168.201.113:9200",
      "192.168.201.114:9200"
    ]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}


```