Prometheus + Grafana + AlertManager 监控体系搭建:Docker 一把梭 摘要基于 Docker 一把梭搭建 Prometheus Grafana AlertManager PrometheusAlert 完整监控告警体系覆盖主机、Redis、MySQL、ES、Kafka 等 15 种组件的指标采集与可视化打通告警路由、抑制去重到飞书、钉钉、企微通知附全部 docker run 命令与 Grafana 看板 ID四个容器半小时落地。服务器挂了你不知道、接口慢了没人发现、Redis 被打爆了才后知后觉……这些问题的根源只有一个没有监控。本文基于 Docker手把手搭建一套完整的监控告警体系Prometheus指标采集 存储 AlertManager告警路由 PrometheusAlert飞书/钉钉通知 Grafana看板所有命令可直接复制执行。一、架构总览组件作用端口Prometheus指标采集、存储、告警规则评估9090AlertManager告警路由、分组、抑制、静默9093PrometheusAlert将告警转发到飞书/钉钉/企微8087Grafana数据可视化看板3000二、部署 Prometheus1. 创建目录mkdir-p/data/prometheus/{config,data,rules}# Prometheus 容器内以 nobody(65534) 运行chown-R65534:65534 /data/prometheus/data2. 配置文件创建/data/prometheus/config/prometheus.ymlglobal:scrape_interval:15s# 告警配置alerting:alertmanagers:-static_configs:-targets:-10.0.1.1:9093# 告警规则文件rule_files:-/prometheus/rules/*.yml# 采集配置scrape_configs:# ---- Prometheus 自身 -----job_name:prometheusstatic_configs:-targets:[10.0.1.1:9090]# ---- 应用监控Spring Boot Actuator -----job_name:appscrape_interval:1mmetrics_path:/actuator/prometheusstatic_configs:-targets:[10.0.1.2:7000,10.0.1.2:8000]# ---- 主机监控Node Exporter -----job_name:node-exporterstatic_configs:-targets:[10.0.1.1:9100]labels:instance:mw-servernodename:mw-server-targets:[10.0.1.2:9100]labels:instance:api-servernodename:api-server-targets:[10.0.1.3:9100]labels:instance:bd-servernodename:bd-server# ---- Redis Cluster每个节点一个 Exporter -----job_name:redis-cluster-1static_configs:-targets:[10.0.1.1:9121]labels:{instance:redis1:6379}-job_name:redis-cluster-2static_configs:-targets:[10.0.1.1:9122]labels:{instance:redis2:6380}-job_name:redis-cluster-3static_configs:-targets:[10.0.1.1:9123]labels:{instance:redis3:6381}# ... 根据实际节点数继续添加# ---- MySQL -----job_name:mysqlstatic_configs:-targets:[10.0.1.1:9104]labels:{instance:master}# ---- MongoDB -----job_name:mongodbstatic_configs:-targets:[10.0.1.1:9220]labels:{instance:mongo-primary}# ---- Elasticsearch -----job_name:elasticsearchscrape_interval:60sscrape_timeout:30sstatic_configs:-targets:[10.0.1.1:9114]# ---- ClickHouse -----job_name:clickhousestatic_configs:-targets:[10.0.1.3:9363]# ---- RocketMQ -----job_name:rocketmqstatic_configs:-targets:[10.0.1.1:5557]# ---- Kafka -----job_name:kafka-exporterstatic_configs:-targets:[10.0.1.3:9308]# ---- ZooKeeper -----job_name:zookeeperstatic_configs:-targets:[10.0.1.1:9141]metric_relabel_configs:-source_labels:[zk_host]target_label:instance# ---- Flink -----job_name:flink-jmstatic_configs:-targets:[10.0.1.3:9020]labels:{group:jm}-job_name:flink-tmstatic_configs:-targets:[10.0.1.3:9021]labels:{group:tm}# ---- DolphinScheduler -----job_name:dolphinschedulermetrics_path:/dolphinscheduler/actuator/prometheusstatic_configs:-targets:[10.0.1.3:12345]labels:{service:dolphinscheduler}metric_relabel_configs:-source_labels:[application]target_label:app-regex:applicationaction:labeldrop# ---- Jenkins -----job_name:jenkinsmetrics_path:/prometheus/static_configs:-targets:[10.0.1.4:12123]# ---- AlertManager 自身 -----job_name:alertmanagerstatic_configs:-targets:[10.0.1.1:9093]# ---- PrometheusAlert 自身 -----job_name:prometheusalertstatic_configs:-targets:[10.0.1.1:8087]这份配置覆盖了15 种组件的监控按需保留或删除即可。各 Exporter 的部署方式不在本文展开每个都是一条docker run可参考文末系列文章中各中间件的专项搭建指南。3. 告警规则示例创建/data/prometheus/rules/basic-rules.ymlgroups:-name:基础告警rules:-alert:CPU 使用率过高expr:100-(avg by (instance)(irate(node_cpu_seconds_total{modeidle}[5m]))) * 10090for:1mlabels:severity:warningannotations:summary:{{ $labels.instance }} CPU 使用率 {{ printf \%.1f\ $value }}%description:CPU 使用率超过 90%持续 1 分钟更多告警规则可参考系列文章中 Redis、ZooKeeper、RocketMQ 的专项告警配置。4. 启动 Prometheusdockerrun-d\--nameprometheus\-p9090:9090\--restartalways\-v/data/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml\-v/data/prometheus/data:/prometheus/data\-v/data/prometheus/rules:/prometheus/rules\prom/prometheus:v3.5.0\--config.file/etc/prometheus/prometheus.yml\--storage.tsdb.path/prometheus/data\--web.enable-lifecycle验证# 访问 Web UIcurlhttp://localhost:9090/-/healthy# 输出 Prometheus Server is Healthy# 查看已注册的 targetscurl-shttp://localhost:9090/api/v1/targets|python3-mjson.tool|head-20--web.enable-lifecycle开启后可以通过curl -X POST http://localhost:9090/-/reload热加载配置不用重启容器。三、部署 AlertManagerAlertManager 负责将 Prometheus 触发的告警进行分组、路由、抑制然后转发给通知渠道。1. 配置文件创建/data/prometheus/config/alertmanager.ymlglobal:resolve_timeout:5mroute:receiver:default-receivergroup_by:[alertname,job,severity]group_wait:30s# 新分组首次发送前等待group_interval:5m# 同一分组内告警发送间隔repeat_interval:3h# 重复告警最小间隔防刷屏routes:# critical 级别单独处理缩短重复间隔-match:severity:criticalreceiver:notify-criticalrepeat_interval:1hreceivers:# 默认接收器通过 PrometheusAlert 转发到飞书-name:default-receiverwebhook_configs:-url:http://10.0.1.1:8087/prometheusalert?typefstplprometheus-fsfsurlhttps://open.feishu.cn/open-apis/bot/v2/hook/YOUR_WEBHOOK_TOKENatallsend_resolved:true# Critical 告警同样走 PrometheusAlert缩短重复间隔-name:notify-criticalwebhook_configs:-url:http://10.0.1.1:8087/prometheusalert?typefstplprometheus-fsfsurlhttps://open.feishu.cn/open-apis/bot/v2/hook/YOUR_WEBHOOK_TOKENatallsend_resolved:true# 抑制规则同一 instance 出现 critical 时抑制 warninginhibit_rules:-source_match:severity:criticaltarget_match:severity:warningequal:[instance]2. 启动dockerrun-d\--namealertmanager\-p9093:9093\--restartalways\-v/data/prometheus/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml\prom/alertmanager\--config.file/etc/alertmanager/alertmanager.yml\--web.external-urlhttp://10.0.1.1:9093验证curlhttp://localhost:9093/-/healthy四、部署 PrometheusAlert飞书/钉钉通知AlertManager 原生只支持邮件和 Webhook要发飞书/钉钉/企微需要一个消息转发中间件。PrometheusAlert 就是干这个的。1. 启动# 先启动一次拷贝 db 文件模板数据dockerrun-d\--nameprometheusalert-init\-p8087:8080\-ePA_LOGIN_USERadmin\-ePA_LOGIN_PASSWORDYourPassword\-ePA_TITLEPrometheus告警\-ePA_OPEN_FEISHU1\-ePA_FSURLhttps://open.feishu.cn/open-apis/bot/v2/hook/YOUR_WEBHOOK_TOKEN\--restartalways\feiyu563/prometheus-alert:latest# 拷贝 db 数据到宿主机实现持久化mkdir-p/data/PrometheusAlertdockercpprometheusalert-init:/app/db /data/PrometheusAlert/dbdockerrm-fprometheusalert-init# 正式启动挂载 db 目录dockerrun-d\--nameprometheusalert\-p8087:8080\-ePA_LOGIN_USERadmin\-ePA_LOGIN_PASSWORDYourPassword\-ePA_TITLEPrometheus告警\-ePA_OPEN_FEISHU1\-ePA_FSURLhttps://open.feishu.cn/open-apis/bot/v2/hook/YOUR_WEBHOOK_TOKEN\-v/data/PrometheusAlert/db:/app/db\--restartalways\feiyu563/prometheus-alert:latest2. 验证访问http://10.0.1.1:8087用 admin / YourPassword 登录可以管理告警模板。告警实例五、部署 Grafana1. 创建目录mkdir-p/data/grafana/data# Grafana 容器内以 uid 472 运行chown-R472:472 /data/grafana/data2. 启动dockerrun-d\--namegrafana\-p3000:3000\--restartalways\-eTZAsia/Shanghai\-eGF_SECURITY_ADMIN_PASSWORDYourPassword\-v/data/grafana/data:/var/lib/grafana\grafana/grafana:latest3. 配置数据源访问http://10.0.1.1:3000用admin/YourPassword登录左侧菜单 → Connections → Data sources → Add data source选择PrometheusURL 填http://10.0.1.1:9090点击Save Test4. 导入看板推荐的 Dashboard ID监控目标Dashboard ID说明主机Node Exporter1860最经典的主机监控看板Redis11835Redis 集群看板MySQL14057MySQL 看板MongoDB12079MongoDB 看板Elasticsearch2322ES 集群看板RocketMQ14612RocketMQ 看板ClickHouse13606ClickHouse 看板Spring Boot4701应用 JVM HTTP 看板Kafka21078Kafka 看板Flink14911Flink 看板DolphinScheduler24841调度器看板导入方式仪表板 → 新建 →导入→ 输入 Dashboard ID → 加载 → 选择 Prometheus 数据源 → Import。实例六、完整部署清单步骤组件命令验证1Prometheusdocker run ... prom/prometheus:v3.5.0curl localhost:9090/-/healthy2AlertManagerdocker run ... prom/alertmanagercurl localhost:9093/-/healthy3PrometheusAlertdocker run ... feiyu563/prometheus-alert访问localhost:80874Grafanadocker run ... grafana/grafana访问localhost:3000四个容器四条命令半小时搞定一套完整的监控告警体系。部署完只是起点真正有用的是告警规则。各中间件的专项告警规则可参考Redis 集群从裸奔到全副武装搭建、可视化、监控、告警、看板一条龙ZooKeeper 从裸奔到全副武装搭建、可视化、监控、告警、看板一条龙RocketMQ 5.x 从裸奔到全副武装监控、告警、看板一条龙MinIO 监控一条龙看板全是「无数据」Prometheus Grafana 从搭建到告警Flink 1.20 基于 Docker 单机部署实战指南ClickHouse 25.4 基于 Docker 单机部署实战指南DolphinScheduler 挂了没人知道Prometheus Grafana 监控兜底方案亲测可用