Nightingale 集成 EMQX基于 Dashboard API 的 MQTT 集群指标采集与告警实战【免费下载链接】nightingaleNightingale is to monitoring and alerting what Grafana is to visualization.项目地址: https://gitcode.com/GitHub_Trending/ni/nightingaleEMQX 插件是 Nightingale 生态中用于监控 EMQX 分布式 MQTT 消息中间件的内置集成它不依赖额外的 Agent 协议而是直接调用 EMQX 自身的 Dashboard REST API 拉取节点状态、集群统计与消息指标随后把指标送入 Nightingale 进行可视化与告警。读完本文你将掌握input.emqx插件的完整配置方法、三大 API 与指标名之间的映射规律、仓库自带告警规则的用法以及 EMQX 5.X 下更轻量的 Prometheus 原生采集方案。一、插件原理与整体工作方式从源码结构与目录组织看Nightingale 的每个集成组件都遵循统一布局integrations/组件/下包含collect/categraf 采集配置、alerts/内置告警规则、markdown/说明文档、i18n/国际化词条与icon/图标。EMQX 组件同样如此其采集配置见 integrations/EMQX/collect/emqx/emqx.toml告警规则见 integrations/EMQX/alerts/emqx_by_categraf.json。插件的数据流可以概括为三步配置实例在 categraf 的input.emqx下配置 EMQX Dashboard 地址、版本与鉴权信息轮询 API采集器按interval周期依次调用/api/${version}/nodes、/api/${version}/stats、/api/${version}/nodes/${node}/metrics三个 REST 接口转换上报把接口返回的 JSON 字段名中的.点替换为_下划线并统一加上emqx_前缀形成 Prometheus 风格的时序指标上报给 Nightingale 的数据源。指标命名遵循一个非常直观的规律API 字段名中的点号即指标名中的下划线。例如connections.count会变成emqx_connections_countpackets.publish.received会变成emqx_packets_publish_received。掌握这条规律后面对 EMQX 5.X 中更丰富的字段也能自行推断指标名。二、插件配置详解插件通过 EMQX 的 Dashboard API 获取指标核心配置如下来自 integrations/EMQX/markdown/README.md 与 integrations/EMQX/collect/emqx/emqx.toml# # 采集周期, 这里指定的 interval 会覆盖 config.toml 中的 interval # interval 60 [[instances]] ## emqx 的 dashboard 地址ip:port, 单个集群监控使用逗号分割多个 IP, 会随机挑一个可用 ip 进行连接请求 ## 多个集群拆分到多个 instance 配置 addresshttp://192.168.11.11:18083,http://192.168.11.12:18083 ## 指定 emqx 的版本支持 v3/v4/v5, 分别对应 emqx 的 3.X/4.X/5.X 版本 versionv3 ## api 需要鉴权输入用户名和密码 usernameadmin passwordpublic配置参数说明参数类型必填说明interval整数秒否采集周期注释掉则沿用 categraf 全局config.toml中的interval显式配置会覆盖全局值address字符串是EMQX Dashboard 地址格式为ip:port监控单个集群时可用逗号分隔多个节点 IP插件会随机挑选一个可用 IP 发起连接请求天然具备简单的故障转移能力version字符串是EMQX 版本号支持v3/v4/v5分别对应 EMQX 3.X / 4.X / 5.X决定 API 路径中的版本段username字符串是Dashboard API 鉴权用户名EMQX 默认管理员为adminpassword字符串是对应密码EMQX 默认密码为public两个重要的拓扑实践单个集群多节点把集群内所有 Dashboard 地址用逗号拼进同一个address插件会自动随机挑一个可用 IP 请求。这样即使某个节点的 Dashboard 不可用采集也不会中断。多个集群一个[[instances]]对应一个集群监控多个 EMQX 集群时应拆分成多个[[instances]]配置块每个实例单独配置地址、版本与凭据。三、三大 API 与指标映射全解3.1/api/${version}/nodes节点级基础指标插件先调用/api/${version}/nodes获取每个节点的版本、运行状态、连接数、负载、内存、进程数与运行时长等信息。接口返回data数组每个元素对应一个节点{ code: 0, data: [ { connections: 0, load1: 0.33, load15: 0.37, load5: 0.36, max_fds: 1048576, memory_total: 191.77M, memory_used: 129.38M, name: emqxnode2.emqx.io, node: emqxnode2.emqx.io, node_status: Running, otp_release: R21/10.3.5.6, process_available: 2097152, process_used: 509, uptime: 3 hours, 12 minutes, 25 seconds, version: 3.2.6 } ] }对应生成的节点级指标如下node标签取自返回中的节点名emqx_node_connections{nodexxx} 10 emqx_node_status{nodexxxx} 1 # 1表示running emqx_node_info{nodexxxx, namexxxx, version3.2.6, otp_releaseR21/10.3.5.6} 1 emqx_node_load1{nodexxxx} 0.10 emqx_node_load5{nodexxxx} 0.37 emqx_node_load15{nodexxxx} 0.32 emqx_node_max_fds{nodexxxx} 1048576 emqx_node_memory_total_bytes{nodexxxx} 194938 emqx_node_memory_used_bytes{nodexxxx} 132956 emqx_node_process_available{nodexxxx} 2097152 emqx_node_process_used{nodexxxx} 508 emqx_node_uptime_seconds{nodexxxx} 1234567 #单位秒几个值得注意的点emqx_node_status是 0/1 布尔型指标1 表示节点 running这是节点存活告警的直接依据emqx_node_memory_total_bytes/emqx_node_memory_used_bytes把接口返回的191.77M这类可读字符串统一换算为字节便于计算内存使用率emqx_node_info是一个携带version、otp_release等版本信息的常量指标值恒为 1可用于按版本维度统计与过滤emqx_node_uptime_seconds已换算为秒可用于检测节点是否发生过近期重启。此外插件会根据所有节点的状态额外生成两个集群级指标cluster标签的值就是配置中的address分别表示当前集群中有多少个节点正常运行、多少个节点已停止emqx_cluster_node_running{clusterxxx} 2 emqx_cluster_node_stopped{clusterxxx} 03.2/api/${version}/stats集群统计指标插件调用/api/${version}/stats获取每个节点的统计信息覆盖连接、订阅、主题、会话、保留消息、规则与路由等维度。接口返回data数组每个元素以node字段开头其后是xxx.count/xxx.max成对出现的统计字段{ code: 0, data: [ { node: emqxnode2.emqx.io, subscriptions.shared.max: 0, subscriptions.max: 0, subscribers.max: 0, resources.max: 0, topics.count: 0, subscriptions.count: 0, suboptions.max: 0, topics.max: 0, sessions.persistent.max: 0, connections.max: 0, sessions.persistent.count: 0, actions.count: 5, retained.count: 5, rules.count: 0, routes.count: 0, subscriptions.shared.count: 0, suboptions.count: 0, sessions.count: 0, actions.max: 5, retained.max: 5, sessions.max: 0, rules.max: 0, routes.max: 0, resources.count: 0, subscribers.count: 0, connections.count: 0 } ] }根据返回的数据生成如下指标count表示当前值max表示历史峰值/上限emqx_subscriptions_shared_max{nodexxx} 123 emqx_subscriptions_max{nodexxx} 123 emqx_subscribers_max{nodexxx} 123 emqx_resources_max{nodexxx} 123 emqx_topics_count{nodexxx} 0 emqx_subscriptions_count{nodexxx} 0 emqx_suboptions_max{nodexxx} 0 emqx_topics_max{nodexxx} 0 emqx_sessions_persistent_max{nodexxx} 0 emqx_connections_max{nodexxx} 0 emqx_sessions_persistent_count{nodexxx} 0 emqx_actions_count{nodexxx} 5 emqx_retained_count{nodexxx} 5 emqx_rules_count{nodexxx} 0 emqx_routes_count{nodexxx} 0 emqx_subscriptions_shared_count{nodexxx} 0 emqx_suboptions_count{nodexxx} 0 emqx_sessions_count{nodexxx} 0 emqx_actions_max{nodexxx} 5 emqx_retained_max{nodexxx} 5 emqx_sessions_max{nodexxx} 0 emqx_rules_max{nodexxx} 0 emqx_routes_max{nodexxx} 0 emqx_resources_count{nodexxx} 0 emqx_subscribers_count{nodexxx} 0 emqx_connections_count{nodexxx} 0这套指标的价值在于容量评估connections、sessions、topics、subscriptions的count与max配对后可以判断当前集群负载离历史峰值还有多大余量是规划扩容的客观依据。3.3/api/${version}/nodes/${node}/metrics消息与报文级指标插件进一步调用/api/${version}/nodes/${node}/metrics获取每个节点细粒度的消息与 MQTT 报文指标包括消息收发、各类型报文CONNECT、PUBLISH、SUBSCRIBE、PINGREQ 等的收发与错误计数、认证情况、规则引擎动作成功率等{ code: 0, data: { rules.matched: 0, messages.sent: 0, packets.disconnect.sent: 0, bytes.sent: 0, packets.disconnect.received: 0, packets.pingresp.sent: 0, packets.pingreq.received: 0, packets.unsubscribe.received: 0, packets.pubcomp.missed: 0, packets.puback.missed: 0, packets.pubcomp.sent: 0, packets.pubcomp.received: 0, packets.pubrec.missed: 0, auth.mqtt.anonymous: 0, packets.connack.auth_error: 0, actions.failure: 0, packets.suback.sent: 0, packets.puback.sent: 0, messages.retained: 5, messages.received: 0, packets.connect.received: 0, messages.forward: 0, packets.pubrel.missed: 0, packets.publish.received: 0, packets.connack.sent: 0, packets.subscribe.received: 0, packets.pubrel.received: 0, packets.pubrec.received: 0, packets.puback.received: 0, packets.sent: 0, packets.received: 0, bytes.received: 0, messages.expired: 0, messages.dropped: 0, messages.qos2.dropped: 0, messages.qos2.expired: 0, packets.pubrel.sent: 0, packets.pubrec.sent: 0, packets.publish.sent: 0, actions.success: 0, packets.publish.error: 0, packets.unsubscribe.error: 0, messages.qos2.received: 0, messages.qos1.received: 0, messages.qos0.received: 0, packets.auth.sent: 0, messages.qos2.sent: 0, messages.qos1.sent: 0, messages.qos0.sent: 0, packets.auth.received: 0, packets.unsuback.sent: 0, packets.connack.error: 0, packets.publish.auth_error: 0, packets.subscribe.error: 0, packets.subscribe.auth_error: 0 } }对应生成的指标如下emqx_rules_matched{nodexxx} 0 emqx_messages_sent{nodexxx} 0 emqx_packets_disconnect_sent{nodexxx} 0 emqx_bytes_sent{nodexxx} 0 emqx_packets_disconnect_received{nodexxx} 0 emqx_packets_pingresp_sent{nodexxx} 0 emqx_packets_pingreq_received{nodexxx} 0 emqx_packets_unsubscribe_received{nodexxx} 0 emqx_packets_pubcomp_missed{nodexxx} 0 emqx_packets_puback_missed{nodexxx} 0 emqx_packets_pubcomp_sent{nodexxx} 0 emqx_packets_pubcomp_received{nodexxx} 0 emqx_packets_pubrec_missed{nodexxx} 0 emqx_auth_mqtt_anonymous{nodexxx} 0 emqx_packets_connack_auth_error{nodexxx} 0 emqx_actions_failure{nodexxx} 0 emqx_packets_suback_sent{nodexxx} 0 emqx_packets_puback_sent{nodexxx} 0 emqx_messages_retained{nodexxx} 5 emqx_messages_received{nodexxx} 0 emqx_packets_connect_received{nodexxx} 0 emqx_messages_forward{nodexxx} 0 emqx_packets_pubrel_missed{nodexxx} 0 emqx_packets_publish_received{nodexxx} 0 emqx_packets_connack_sent{nodexxx} 0 emqx_packets_subscribe_received{nodexxx} 0 emqx_packets_pubrel_received{nodexxx} 0 emqx_packets_pubrec_received{nodexxx} 0 emqx_packets_puback_received{nodexxx} 0 emqx_packets_sent{nodexxx} 0 emqx_packets_received{nodexxx} 0 emqx_bytes_received{nodexxx} 0 emqx_messages_expired{nodexxx} 0 emqx_messages_dropped{nodexxx} 0 emqx_messages_qos2_dropped{nodexxx} 0 emqx_messages_qos2_expired{nodexxx} 0 emqx_packets_pubrel_sent{nodexxx} 0 emqx_packets_pubrec_sent{nodexxx} 0 emqx_packets_publish_sent{nodexxx} 0 emqx_actions_success{nodexxx} 0 emqx_packets_publish_error{nodexxx} 0 emqx_packets_unsubscribe_error{nodexxx} 0 emqx_messages_qos2_received{nodexxx} 0 emqx_messages_qos1_received{nodexxx} 0 emqx_messages_qos0_received{nodexxx} 0 emqx_packets_auth_sent{nodexxx} 0 emqx_messages_qos2_sent{nodexxx} 0 emqx_messages_qos1_sent{nodexxx} 0 emqx_messages_qos0_sent{nodexxx} 0 emqx_packets_auth_received{nodexxx} 0 emqx_packets_unsuback_sent{nodexxx} 0 emqx_packets_connack_error{nodexxx} 0 emqx_packets_publish_auth_error{nodexxx} 0 emqx_packets_subscribe_error{nodexxx} 0 emqx_packets_subscribe_auth_error{nodexxx} 0这一组指标刻画了消息管道与协议层的健康状况。特别值得关注的是emqx_messages_dropped消息丢弃、emqx_packets_connack_auth_error连接认证失败、emqx_actions_failure/emqx_actions_success规则引擎动作成败——它们也是仓库内置告警规则的直接依赖项见下一节。由于这些是计数器指标在告警中应使用rate()计算速率而非直接比较原始值。四、仓库内置告警规则实战Nightingale 为 EMQX 组件预置了 9 条开箱即用的告警规则全部定义在 integrations/EMQX/alerts/emqx_by_categraf.json 中类型为prometheus、生产类别metric可直接导入 Prometheus 类数据源使用。规则清单如下告警名称PromQL级别severity持续时长prom_for_durationEMQX 节点不在运行状态emqx_node_status ! 11P160sEMQX 集群存在已停止的节点emqx_cluster_node_stopped 01P1120sEMQX 连接数逼近文件描述符上限emqx_node_connections / (emqx_node_max_fds 0) * 100 802300sEMQX Erlang 进程数逼近上限emqx_node_process_used / (emqx_node_process_available 0) * 100 802300sEMQX 节点内存使用率过高emqx_node_memory_used_bytes / (emqx_node_memory_total_bytes 0) * 100 852300sEMQX 消息丢弃激增rate(emqx_messages_dropped[5m]) 12300sEMQX 客户端认证失败激增rate(emqx_packets_connack_auth_error[5m]) 12300sEMQX 规则引擎动作失败率过高rate(emqx_actions_failure[5m]) / ((rate(emqx_actions_success[5m]) rate(emqx_actions_failure[5m])) 0) * 100 52300sEMQX 节点近期发生过重启emqx_node_uptime_seconds 3003P360s这些规则的默认配置prom_eval_interval15 秒、notify_repeat_step60 秒、notify_recovered开启、enable_stime/enable_etime全天 00:00–23:59、disabled为 1在导入后可根据业务调整每条规则都带有append_tags形式的告警标识如alertnameEmqxNodeNotRunning、alertnameEmqxClusterNodeStopped便于在 Nightingale 中做告警聚合与路由。更难得的是每条规则都内置了完整的排查 Runbookannotations.action字段例如节点不在运行状态登录节点执行emqx ctl status与systemctl status emqx确认进程进程存活但状态异常的查看log/emqx.log.1尾部常见为 Erlang 虚拟机内存耗尽或磁盘写满触发节点自保进程不在则先df -h、free -g排除资源问题再启动恢复后执行emqx ctl cluster status确认节点重新加入集群并观察连接数回流。连接数逼近 fd 上限执行emqx ctl listeners查看各监听器连接配置用cat /proc/$(pgrep -f beam.smp)/limits | grep open files确认实际 fd 上限调大 systemd 的LimitNOFILE与 emqx.conf 的node.max_ports并重启注意评估内存每连接约占几十 KB同时在网关侧限制异常客户端疯狂重连。Erlang 进程数逼近上限emqx ctl vm查看 process/used 与 process/limit 实时值区分正常业务增长与会话泄漏大量clean_sessionfalse的持久会话累积必要时调大node.process_limit。内存使用率过高emqx ctl vm memory查看 processes/binary/ets 分布——binary 占比高多为消息积压于会话队列应急可调低mqueue_max_len限制单会话队列长度避免个别慢消费者拖垮节点。消息丢弃激增emqx ctl metrics | grep dropped区分丢弃原因queue_full / no_subscribers / expired后分别处理。认证失败激增查看log/emqx.log中失败的 clientid 与来源 IP 分布区分认证后端故障与暴力破解后者在网关侧限流并开黑名单。规则引擎动作失败率过高在 Dashboard 规则引擎页面定位失败规则区分下游不可达、超时还是数据格式不匹配。节点近期重启先排除计划内发布/扩缩容再查journalctl -u emqx --since -30min与dmesg -T | grep -i killed process排除 OOM重启后对设备重连做限速避免重连风暴。这些规则的中英文文案由 integrations/EMQX/i18n/en_US.json 提供英文环境下展示的告警名称与 Runbook 均为英文译文。五、EMQX 5.X 的 Prometheus 原生采集方案原文档特别指出5.X 的 EMQX 提供了原生的 Prometheus 接口可以完全不使用上述 Dashboard API 插件方案而是直接用 categraf 的input.prometheus插件采集http://ip:18083/api/v5/prometheus/stats这条路径相比 Dashboard API 方案有两个明显优势一是无需额外配置用户名密码认证部署更简单二是原生暴露的即为 Prometheus 文本格式指标语义与_count/_max等命名更统一配合 Prometheus 生态的 relabel 与 recording rule 也更顺手。需要注意的适用前提该方案仅适用于EMQX 5.X3.X / 4.X 没有此接口必须使用本文前几章的 Dashboard API 插件version分别设为v3/v4若集群存在多个节点需要为每个节点各配置一个input.prometheus采集目标Dashboard API 方案则只需在address中逗号分隔多 IP 即可自动容错或用服务发现机制自动发现节点input.prometheus与input.emqx两种方式产出的指标命名不同告警规则需对应调整不能混用。六、集成资源在 Nightingale 中的落地方式最后从源码角度说明这些集成资源是如何进入 Nightingale 运行时的内置集成初始化center/integration/init.go 会在启动时扫描integrations/目录。对每个组件依次装载markdown/下的 README含README.en_US.md等多语言副本、alerts/*.json解析为告警规则并写入builtin_payloads、dashboards/、metrics/等资源并把icon/下的图标注册为组件 Logo供集成中心页面展示与一键导入。若设置了disable_integration_init配置项则跳过该初始化过程。AI 文档检索索引aiagent/tools/integrations_loader.go 会把每个组件的markdown/README.md转成[integration-doc]条目、collect/*/*.toml转成[integration-config]条目并入文档索引使 AI 助手通过search_n9e_docs检索时能直接命中真实的[[instances]]写法——这意味着本文介绍的配置样例不仅是人读的文档也是 AI 配置助手的事实依据。采集端落地采集配置最终由 categraf 加载执行指标经 Nightingale 的 Prometheus 数据源入库后即可在仪表盘中绘图并配合上一节的告警规则实现闭环监控。至此从「categraf 拉取 EMQX Dashboard API」到「指标入库」再到「内置告警 排障 Runbook」的完整链路已经打通EMQX 3.X/4.X 用input.emqx插件按本文配置5.X 可按需切换到input.prometheus原生方案两层手段共同覆盖了 MQTT 集群的全生命周期监控需求。【免费下载链接】nightingaleNightingale is to monitoring and alerting what Grafana is to visualization.项目地址: https://gitcode.com/GitHub_Trending/ni/nightingale创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 
长河君 · 2026/9/15 22:29:00 · 阅读 1.2k Nightingale 集成 EMQX：基于 Dashboard API 的 MQTT 集群指标采集与告警实战
Nightingale 集成 EMQX基于 Dashboard API 的 MQTT 集群指标采集与告警实战【免费下载链接】nightingaleNightingale is to monitoring and alerting what Grafana is to visualization.项目地址: https://gitcode.com/GitHub_Trending/ni/nightingaleEMQX 插件是 Nightingale 生态中用于监控 EMQX 分布式 MQTT 消息中间件的内置集成它不依赖额外的 Agent 协议而是直接调用 EMQX 自身的 Dashboard REST API 拉取节点状态、集群统计与消息指标随后把指标送入 Nightingale 进行可视化与告警。读完本文你将掌握input.emqx插件的完整配置方法、三大 API 与指标名之间的映射规律、仓库自带告警规则的用法以及 EMQX 5.X 下更轻量的 Prometheus 原生采集方案。一、插件原理与整体工作方式从源码结构与目录组织看Nightingale 的每个集成组件都遵循统一布局integrations/组件/下包含collect/categraf 采集配置、alerts/内置告警规则、markdown/说明文档、i18n/国际化词条与icon/图标。EMQX 组件同样如此其采集配置见 integrations/EMQX/collect/emqx/emqx.toml告警规则见 integrations/EMQX/alerts/emqx_by_categraf.json。插件的数据流可以概括为三步配置实例在 categraf 的input.emqx下配置 EMQX Dashboard 地址、版本与鉴权信息轮询 API采集器按interval周期依次调用/api/${version}/nodes、/api/${version}/stats、/api/${version}/nodes/${node}/metrics三个 REST 接口转换上报把接口返回的 JSON 字段名中的.点替换为_下划线并统一加上emqx_前缀形成 Prometheus 风格的时序指标上报给 Nightingale 的数据源。指标命名遵循一个非常直观的规律API 字段名中的点号即指标名中的下划线。例如connections.count会变成emqx_connections_countpackets.publish.received会变成emqx_packets_publish_received。掌握这条规律后面对 EMQX 5.X 中更丰富的字段也能自行推断指标名。二、插件配置详解插件通过 EMQX 的 Dashboard API 获取指标核心配置如下来自 integrations/EMQX/markdown/README.md 与 integrations/EMQX/collect/emqx/emqx.toml# # 采集周期, 这里指定的 interval 会覆盖 config.toml 中的 interval # interval 60 [[instances]] ## emqx 的 dashboard 地址ip:port, 单个集群监控使用逗号分割多个 IP, 会随机挑一个可用 ip 进行连接请求 ## 多个集群拆分到多个 instance 配置 addresshttp://192.168.11.11:18083,http://192.168.11.12:18083 ## 指定 emqx 的版本支持 v3/v4/v5, 分别对应 emqx 的 3.X/4.X/5.X 版本 versionv3 ## api 需要鉴权输入用户名和密码 usernameadmin passwordpublic配置参数说明参数类型必填说明interval整数秒否采集周期注释掉则沿用 categraf 全局config.toml中的interval显式配置会覆盖全局值address字符串是EMQX Dashboard 地址格式为ip:port监控单个集群时可用逗号分隔多个节点 IP插件会随机挑选一个可用 IP 发起连接请求天然具备简单的故障转移能力version字符串是EMQX 版本号支持v3/v4/v5分别对应 EMQX 3.X / 4.X / 5.X决定 API 路径中的版本段username字符串是Dashboard API 鉴权用户名EMQX 默认管理员为adminpassword字符串是对应密码EMQX 默认密码为public两个重要的拓扑实践单个集群多节点把集群内所有 Dashboard 地址用逗号拼进同一个address插件会自动随机挑一个可用 IP 请求。这样即使某个节点的 Dashboard 不可用采集也不会中断。多个集群一个[[instances]]对应一个集群监控多个 EMQX 集群时应拆分成多个[[instances]]配置块每个实例单独配置地址、版本与凭据。三、三大 API 与指标映射全解3.1/api/${version}/nodes节点级基础指标插件先调用/api/${version}/nodes获取每个节点的版本、运行状态、连接数、负载、内存、进程数与运行时长等信息。接口返回data数组每个元素对应一个节点{ code: 0, data: [ { connections: 0, load1: 0.33, load15: 0.37, load5: 0.36, max_fds: 1048576, memory_total: 191.77M, memory_used: 129.38M, name: emqxnode2.emqx.io, node: emqxnode2.emqx.io, node_status: Running, otp_release: R21/10.3.5.6, process_available: 2097152, process_used: 509, uptime: 3 hours, 12 minutes, 25 seconds, version: 3.2.6 } ] }对应生成的节点级指标如下node标签取自返回中的节点名emqx_node_connections{nodexxx} 10 emqx_node_status{nodexxxx} 1 # 1表示running emqx_node_info{nodexxxx, namexxxx, version3.2.6, otp_releaseR21/10.3.5.6} 1 emqx_node_load1{nodexxxx} 0.10 emqx_node_load5{nodexxxx} 0.37 emqx_node_load15{nodexxxx} 0.32 emqx_node_max_fds{nodexxxx} 1048576 emqx_node_memory_total_bytes{nodexxxx} 194938 emqx_node_memory_used_bytes{nodexxxx} 132956 emqx_node_process_available{nodexxxx} 2097152 emqx_node_process_used{nodexxxx} 508 emqx_node_uptime_seconds{nodexxxx} 1234567 #单位秒几个值得注意的点emqx_node_status是 0/1 布尔型指标1 表示节点 running这是节点存活告警的直接依据emqx_node_memory_total_bytes/emqx_node_memory_used_bytes把接口返回的191.77M这类可读字符串统一换算为字节便于计算内存使用率emqx_node_info是一个携带version、otp_release等版本信息的常量指标值恒为 1可用于按版本维度统计与过滤emqx_node_uptime_seconds已换算为秒可用于检测节点是否发生过近期重启。此外插件会根据所有节点的状态额外生成两个集群级指标cluster标签的值就是配置中的address分别表示当前集群中有多少个节点正常运行、多少个节点已停止emqx_cluster_node_running{clusterxxx} 2 emqx_cluster_node_stopped{clusterxxx} 03.2/api/${version}/stats集群统计指标插件调用/api/${version}/stats获取每个节点的统计信息覆盖连接、订阅、主题、会话、保留消息、规则与路由等维度。接口返回data数组每个元素以node字段开头其后是xxx.count/xxx.max成对出现的统计字段{ code: 0, data: [ { node: emqxnode2.emqx.io, subscriptions.shared.max: 0, subscriptions.max: 0, subscribers.max: 0, resources.max: 0, topics.count: 0, subscriptions.count: 0, suboptions.max: 0, topics.max: 0, sessions.persistent.max: 0, connections.max: 0, sessions.persistent.count: 0, actions.count: 5, retained.count: 5, rules.count: 0, routes.count: 0, subscriptions.shared.count: 0, suboptions.count: 0, sessions.count: 0, actions.max: 5, retained.max: 5, sessions.max: 0, rules.max: 0, routes.max: 0, resources.count: 0, subscribers.count: 0, connections.count: 0 } ] }根据返回的数据生成如下指标count表示当前值max表示历史峰值/上限emqx_subscriptions_shared_max{nodexxx} 123 emqx_subscriptions_max{nodexxx} 123 emqx_subscribers_max{nodexxx} 123 emqx_resources_max{nodexxx} 123 emqx_topics_count{nodexxx} 0 emqx_subscriptions_count{nodexxx} 0 emqx_suboptions_max{nodexxx} 0 emqx_topics_max{nodexxx} 0 emqx_sessions_persistent_max{nodexxx} 0 emqx_connections_max{nodexxx} 0 emqx_sessions_persistent_count{nodexxx} 0 emqx_actions_count{nodexxx} 5 emqx_retained_count{nodexxx} 5 emqx_rules_count{nodexxx} 0 emqx_routes_count{nodexxx} 0 emqx_subscriptions_shared_count{nodexxx} 0 emqx_suboptions_count{nodexxx} 0 emqx_sessions_count{nodexxx} 0 emqx_actions_max{nodexxx} 5 emqx_retained_max{nodexxx} 5 emqx_sessions_max{nodexxx} 0 emqx_rules_max{nodexxx} 0 emqx_routes_max{nodexxx} 0 emqx_resources_count{nodexxx} 0 emqx_subscribers_count{nodexxx} 0 emqx_connections_count{nodexxx} 0这套指标的价值在于容量评估connections、sessions、topics、subscriptions的count与max配对后可以判断当前集群负载离历史峰值还有多大余量是规划扩容的客观依据。3.3/api/${version}/nodes/${node}/metrics消息与报文级指标插件进一步调用/api/${version}/nodes/${node}/metrics获取每个节点细粒度的消息与 MQTT 报文指标包括消息收发、各类型报文CONNECT、PUBLISH、SUBSCRIBE、PINGREQ 等的收发与错误计数、认证情况、规则引擎动作成功率等{ code: 0, data: { rules.matched: 0, messages.sent: 0, packets.disconnect.sent: 0, bytes.sent: 0, packets.disconnect.received: 0, packets.pingresp.sent: 0, packets.pingreq.received: 0, packets.unsubscribe.received: 0, packets.pubcomp.missed: 0, packets.puback.missed: 0, packets.pubcomp.sent: 0, packets.pubcomp.received: 0, packets.pubrec.missed: 0, auth.mqtt.anonymous: 0, packets.connack.auth_error: 0, actions.failure: 0, packets.suback.sent: 0, packets.puback.sent: 0, messages.retained: 5, messages.received: 0, packets.connect.received: 0, messages.forward: 0, packets.pubrel.missed: 0, packets.publish.received: 0, packets.connack.sent: 0, packets.subscribe.received: 0, packets.pubrel.received: 0, packets.pubrec.received: 0, packets.puback.received: 0, packets.sent: 0, packets.received: 0, bytes.received: 0, messages.expired: 0, messages.dropped: 0, messages.qos2.dropped: 0, messages.qos2.expired: 0, packets.pubrel.sent: 0, packets.pubrec.sent: 0, packets.publish.sent: 0, actions.success: 0, packets.publish.error: 0, packets.unsubscribe.error: 0, messages.qos2.received: 0, messages.qos1.received: 0, messages.qos0.received: 0, packets.auth.sent: 0, messages.qos2.sent: 0, messages.qos1.sent: 0, messages.qos0.sent: 0, packets.auth.received: 0, packets.unsuback.sent: 0, packets.connack.error: 0, packets.publish.auth_error: 0, packets.subscribe.error: 0, packets.subscribe.auth_error: 0 } }对应生成的指标如下emqx_rules_matched{nodexxx} 0 emqx_messages_sent{nodexxx} 0 emqx_packets_disconnect_sent{nodexxx} 0 emqx_bytes_sent{nodexxx} 0 emqx_packets_disconnect_received{nodexxx} 0 emqx_packets_pingresp_sent{nodexxx} 0 emqx_packets_pingreq_received{nodexxx} 0 emqx_packets_unsubscribe_received{nodexxx} 0 emqx_packets_pubcomp_missed{nodexxx} 0 emqx_packets_puback_missed{nodexxx} 0 emqx_packets_pubcomp_sent{nodexxx} 0 emqx_packets_pubcomp_received{nodexxx} 0 emqx_packets_pubrec_missed{nodexxx} 0 emqx_auth_mqtt_anonymous{nodexxx} 0 emqx_packets_connack_auth_error{nodexxx} 0 emqx_actions_failure{nodexxx} 0 emqx_packets_suback_sent{nodexxx} 0 emqx_packets_puback_sent{nodexxx} 0 emqx_messages_retained{nodexxx} 5 emqx_messages_received{nodexxx} 0 emqx_packets_connect_received{nodexxx} 0 emqx_messages_forward{nodexxx} 0 emqx_packets_pubrel_missed{nodexxx} 0 emqx_packets_publish_received{nodexxx} 0 emqx_packets_connack_sent{nodexxx} 0 emqx_packets_subscribe_received{nodexxx} 0 emqx_packets_pubrel_received{nodexxx} 0 emqx_packets_pubrec_received{nodexxx} 0 emqx_packets_puback_received{nodexxx} 0 emqx_packets_sent{nodexxx} 0 emqx_packets_received{nodexxx} 0 emqx_bytes_received{nodexxx} 0 emqx_messages_expired{nodexxx} 0 emqx_messages_dropped{nodexxx} 0 emqx_messages_qos2_dropped{nodexxx} 0 emqx_messages_qos2_expired{nodexxx} 0 emqx_packets_pubrel_sent{nodexxx} 0 emqx_packets_pubrec_sent{nodexxx} 0 emqx_packets_publish_sent{nodexxx} 0 emqx_actions_success{nodexxx} 0 emqx_packets_publish_error{nodexxx} 0 emqx_packets_unsubscribe_error{nodexxx} 0 emqx_messages_qos2_received{nodexxx} 0 emqx_messages_qos1_received{nodexxx} 0 emqx_messages_qos0_received{nodexxx} 0 emqx_packets_auth_sent{nodexxx} 0 emqx_messages_qos2_sent{nodexxx} 0 emqx_messages_qos1_sent{nodexxx} 0 emqx_messages_qos0_sent{nodexxx} 0 emqx_packets_auth_received{nodexxx} 0 emqx_packets_unsuback_sent{nodexxx} 0 emqx_packets_connack_error{nodexxx} 0 emqx_packets_publish_auth_error{nodexxx} 0 emqx_packets_subscribe_error{nodexxx} 0 emqx_packets_subscribe_auth_error{nodexxx} 0这一组指标刻画了消息管道与协议层的健康状况。特别值得关注的是emqx_messages_dropped消息丢弃、emqx_packets_connack_auth_error连接认证失败、emqx_actions_failure/emqx_actions_success规则引擎动作成败——它们也是仓库内置告警规则的直接依赖项见下一节。由于这些是计数器指标在告警中应使用rate()计算速率而非直接比较原始值。四、仓库内置告警规则实战Nightingale 为 EMQX 组件预置了 9 条开箱即用的告警规则全部定义在 integrations/EMQX/alerts/emqx_by_categraf.json 中类型为prometheus、生产类别metric可直接导入 Prometheus 类数据源使用。规则清单如下告警名称PromQL级别severity持续时长prom_for_durationEMQX 节点不在运行状态emqx_node_status ! 11P160sEMQX 集群存在已停止的节点emqx_cluster_node_stopped 01P1120sEMQX 连接数逼近文件描述符上限emqx_node_connections / (emqx_node_max_fds 0) * 100 802300sEMQX Erlang 进程数逼近上限emqx_node_process_used / (emqx_node_process_available 0) * 100 802300sEMQX 节点内存使用率过高emqx_node_memory_used_bytes / (emqx_node_memory_total_bytes 0) * 100 852300sEMQX 消息丢弃激增rate(emqx_messages_dropped[5m]) 12300sEMQX 客户端认证失败激增rate(emqx_packets_connack_auth_error[5m]) 12300sEMQX 规则引擎动作失败率过高rate(emqx_actions_failure[5m]) / ((rate(emqx_actions_success[5m]) rate(emqx_actions_failure[5m])) 0) * 100 52300sEMQX 节点近期发生过重启emqx_node_uptime_seconds 3003P360s这些规则的默认配置prom_eval_interval15 秒、notify_repeat_step60 秒、notify_recovered开启、enable_stime/enable_etime全天 00:00–23:59、disabled为 1在导入后可根据业务调整每条规则都带有append_tags形式的告警标识如alertnameEmqxNodeNotRunning、alertnameEmqxClusterNodeStopped便于在 Nightingale 中做告警聚合与路由。更难得的是每条规则都内置了完整的排查 Runbookannotations.action字段例如节点不在运行状态登录节点执行emqx ctl status与systemctl status emqx确认进程进程存活但状态异常的查看log/emqx.log.1尾部常见为 Erlang 虚拟机内存耗尽或磁盘写满触发节点自保进程不在则先df -h、free -g排除资源问题再启动恢复后执行emqx ctl cluster status确认节点重新加入集群并观察连接数回流。连接数逼近 fd 上限执行emqx ctl listeners查看各监听器连接配置用cat /proc/$(pgrep -f beam.smp)/limits | grep open files确认实际 fd 上限调大 systemd 的LimitNOFILE与 emqx.conf 的node.max_ports并重启注意评估内存每连接约占几十 KB同时在网关侧限制异常客户端疯狂重连。Erlang 进程数逼近上限emqx ctl vm查看 process/used 与 process/limit 实时值区分正常业务增长与会话泄漏大量clean_sessionfalse的持久会话累积必要时调大node.process_limit。内存使用率过高emqx ctl vm memory查看 processes/binary/ets 分布——binary 占比高多为消息积压于会话队列应急可调低mqueue_max_len限制单会话队列长度避免个别慢消费者拖垮节点。消息丢弃激增emqx ctl metrics | grep dropped区分丢弃原因queue_full / no_subscribers / expired后分别处理。认证失败激增查看log/emqx.log中失败的 clientid 与来源 IP 分布区分认证后端故障与暴力破解后者在网关侧限流并开黑名单。规则引擎动作失败率过高在 Dashboard 规则引擎页面定位失败规则区分下游不可达、超时还是数据格式不匹配。节点近期重启先排除计划内发布/扩缩容再查journalctl -u emqx --since -30min与dmesg -T | grep -i killed process排除 OOM重启后对设备重连做限速避免重连风暴。这些规则的中英文文案由 integrations/EMQX/i18n/en_US.json 提供英文环境下展示的告警名称与 Runbook 均为英文译文。五、EMQX 5.X 的 Prometheus 原生采集方案原文档特别指出5.X 的 EMQX 提供了原生的 Prometheus 接口可以完全不使用上述 Dashboard API 插件方案而是直接用 categraf 的input.prometheus插件采集http://ip:18083/api/v5/prometheus/stats这条路径相比 Dashboard API 方案有两个明显优势一是无需额外配置用户名密码认证部署更简单二是原生暴露的即为 Prometheus 文本格式指标语义与_count/_max等命名更统一配合 Prometheus 生态的 relabel 与 recording rule 也更顺手。需要注意的适用前提该方案仅适用于EMQX 5.X3.X / 4.X 没有此接口必须使用本文前几章的 Dashboard API 插件version分别设为v3/v4若集群存在多个节点需要为每个节点各配置一个input.prometheus采集目标Dashboard API 方案则只需在address中逗号分隔多 IP 即可自动容错或用服务发现机制自动发现节点input.prometheus与input.emqx两种方式产出的指标命名不同告警规则需对应调整不能混用。六、集成资源在 Nightingale 中的落地方式最后从源码角度说明这些集成资源是如何进入 Nightingale 运行时的内置集成初始化center/integration/init.go 会在启动时扫描integrations/目录。对每个组件依次装载markdown/下的 README含README.en_US.md等多语言副本、alerts/*.json解析为告警规则并写入builtin_payloads、dashboards/、metrics/等资源并把icon/下的图标注册为组件 Logo供集成中心页面展示与一键导入。若设置了disable_integration_init配置项则跳过该初始化过程。AI 文档检索索引aiagent/tools/integrations_loader.go 会把每个组件的markdown/README.md转成[integration-doc]条目、collect/*/*.toml转成[integration-config]条目并入文档索引使 AI 助手通过search_n9e_docs检索时能直接命中真实的[[instances]]写法——这意味着本文介绍的配置样例不仅是人读的文档也是 AI 配置助手的事实依据。采集端落地采集配置最终由 categraf 加载执行指标经 Nightingale 的 Prometheus 数据源入库后即可在仪表盘中绘图并配合上一节的告警规则实现闭环监控。至此从「categraf 拉取 EMQX Dashboard API」到「指标入库」再到「内置告警 排障 Runbook」的完整链路已经打通EMQX 3.X/4.X 用input.emqx插件按本文配置5.X 可按需切换到input.prometheus原生方案两层手段共同覆盖了 MQTT 集群的全生命周期监控需求。【免费下载链接】nightingaleNightingale is to monitoring and alerting what Grafana is to visualization.项目地址: https://gitcode.com/GitHub_Trending/ni/nightingale创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考