Kube-Prometheus 部署基于 kube-prometheus release-0.18包含镜像调整、副本数、持久化、Service 暴露等步骤。1. 下载安装包git clone -b release-0.18 https://github.com/prometheus-operator/kube-prometheus.git cd kube-prometheus/manifests/setup/ kubectl create -f . # 先创建 CRD 等基础资源 cd .. grep -r -i image: . # 查看所有镜像按需修改镜像地址编辑的镜像地址改成自己私有仓库再把导入的镜像tar包或者拉取的镜像打tag重新上传到私有仓库# 先登录私有仓库 docker login reg.westos.org -u admin -p westos # 1. alertmanager docker tag quay.io/prometheus/alertmanager:v0.33.0 reg.westos.org/prometheus/alertmanager:v0.33.0 docker push reg.westos.org/prometheus/alertmanager:v0.33.0 # 2. blackbox-exporter docker tag quay.io/prometheus/blackbox-exporter:v0.28.0 reg.westos.org/prometheus/blackbox-exporter:v0.28.0 docker push reg.westos.org/prometheus/blackbox-exporter:v0.28.0 # 3. kube-state-metrics docker tag registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.0 reg.westos.org/prometheus/kube-state-metrics:v2.19.0 docker push reg.westos.org/prometheus/kube-state-metrics:v2.19.0 # 4. node-exporter docker tag quay.io/prometheus/node-exporter:v1.11.1 reg.westos.org/prometheus/node-exporter:v1.11.1 docker push reg.westos.org/prometheus/node-exporter:v1.11.1 # 5. prometheus docker tag quay.io/prometheus/prometheus:v3.12.0 reg.westos.org/prometheus/prometheus:v3.12.0 docker push reg.westos.org/prometheus/prometheus:v3.12.0 # 6. prometheus-adapter docker tag registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0 reg.westos.org/prometheus/prometheus-adapter:v0.12.0 docker push reg.westos.org/prometheus/prometheus-adapter:v0.12.0 # 7. configmap-reload docker tag ghcr.io/jimmidyson/configmap-reload:v0.15.0 reg.westos.org/prometheus/configmap-reload:v0.15.0 docker push reg.westos.org/prometheus/configmap-reload:v0.15.0 # 8. grafana docker tag grafana/grafana:13.0.2 reg.westos.org/prometheus/grafana:13.0.2 docker push reg.westos.org/prometheus/grafana:13.0.2 # 9. kube-rbac-proxy docker tag quay.io/brancz/kube-rbac-proxy:v0.22.0 reg.westos.org/prometheus/kube-rbac-proxy:v0.22.0 docker push reg.westos.org/prometheus/kube-rbac-proxy:v0.22.0 # 10. prometheus-operator docker tag quay.io/prometheus-operator/prometheus-operator:v0.92.0 reg.westos.org/prometheus/prometheus-operator:v0.92.0 docker push reg.westos.org/prometheus/prometheus-operator:v0.92.02. 修改配置2.1 调整副本数# 修改 alertmanager 副本数为 1 vim alertmanager-alertmanager.yaml # 修改 replicas: 1 # 修改 prometheus 副本数为 1 vim prometheus-prometheus.yaml # 修改 replicas: 13. Grafana 数据持久化3.1 创建 PVCapiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pvc-nfs namespace: monitoring spec: accessModes: - ReadWriteMany # 多节点同时读写 resources: requests: storage: 10Gi # 存储大小 storageClassName: nfs-client # 使用 nfs-client StorageClasskubectl create -f grafana-pvc.yaml kubectl get pvc -n monitoring3.2 修改 Grafana Deploymentvim grafana-deployment.yaml修改 volumes 部分注意缩进yaml volumes: # - emptyDir: {} # name: grafana-storage - name: grafana-storage persistentVolumeClaim: claimName: grafana-pvc-nfs readOnly: false4. Prometheus 持久化配置vim prometheus-prometheus.yaml添加/修改yaml image: reg.westos.org/prometheus/prometheus:v3.5.0 storage: volumeClaimTemplate: spec: accessModes: [ ReadWriteOnce ] storageClassName: nfs-client resources: requests: storage: 30Gi5. 部署 Prometheus 服务kubectl create -f .6. 清除所有网络策略kubectl -n monitoring delete networkpolicies --all清除 NetworkPolicy 方便外部访问生产环境请按需保留。7. 暴露服务为 LoadBalancer7.1 Grafanakubectl get svc grafana -n monitoring kubectl edit svc grafana -n monitoring # 将 type 改为 LoadBalancer kubectl -n monitoring get svc grafana输出示例text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 10.99.135.255 192.168.36.101 3000:44532/TCP 159m访问http://192.168.36.101:3000用户名admin密码admin7.2 Prometheuskubectl get svc prometheus-k8s -n monitoring kubectl edit svc prometheus-k8s -n monitoring # 将 type 改为 LoadBalancer kubectl -n monitoring get svc prometheus-k8s输出示例text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus-k8s LoadBalancer 10.103.213.201 192.168.36.102 9090:33611/TCP,8080:45454/TCP 163m访问http://192.168.36.102:9090云原生应用监控etcd 集群监控在 kube-prometheus 基础上将 etcd 纳入监控。核心创建 Service/Endpoints Secret ServiceMonitor。1. 查看 etcd 证书与端口在 k8s master 节点操作。bash# 查看 etcd 启动参数中的证书路径 grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml输出示例text- --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.keybash# 直接测试 etcd metrics 接口 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.36.152:2379/metrics -k | tail -1 # 输出类似promhttp_metric_handler_requests_total{code503} 0bash# 查看 etcd 监听端口 ss -tunlp | grep etcd2. 创建 etcd Service 与 Endpointsetcd 是静态 Pod没有自动创建 Service需要手动创建 Endpoints 和 Service 供 Prometheus 发现。bashmkdir -p servicemonitor/etcd cd servicemonitor/etcd/ vim etcd-service.ymlyaml--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom # 标签供 Service 和 ServiceMonitor 匹配 name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.36.152 # etcd 节点 IP按实际修改 ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIPbashkubectl apply -f etcd-service.yml kubectl get svc etcd-prom -n kube-system输出示例textNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd-prom ClusterIP 10.100.206.20 none 2379/TCP 128mbash# 通过 Service 测试抓取 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://10.100.206.20:2379/metrics -k | tail -13. 创建 etcd 证书 SecretPrometheus 需要证书才能访问 etcd 的 https metrics。bashkubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key kubectl -n monitoring get secrets etcd-ssl输出示例textNAME TYPE DATA AGE etcd-ssl Opaque 3 130m4. Prometheus 挂载 etcd 证书bashcd /root/kube-prometheus/manifests/ vim prometheus-prometheus.yaml在spec末尾追加yamlsecrets: - etcd-ssl # 将 Secret 挂载到 Prometheus Podbash# 更新资源replace 会整体替换 kubectl replace -f prometheus-prometheus.yaml # 查看证书是否挂载成功 kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- \ ls /etc/prometheus/secrets/etcd-ssl5. 创建 ServiceMonitor让 Prometheus 自动发现并抓取 etcd 指标。bashcd - vim etcd-servicemonitor.yamlyaml--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd spec: jobLabel: k8s-etcd-cluster-apps endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: https # 使用 https 协议 tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true # 关闭证书校验测试环境方便 selector: matchLabels: app: etcd-prom # 匹配目标 Service 的标签 namespaceSelector: matchNames: - kube-system # 目标 Service 所在命名空间bashkubectl create -f etcd-servicemonitor.yaml kubectl get servicemonitor -n monitoring | grep etcd6. 查看监控数据与 Grafana 面板在 Prometheus UI 中查询 etcd 指标确认 Target 为 UP。Grafana 导入 etcd 监控面板texthttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/云原生应用监控etcd 集群监控笔记在 kube-prometheus 基础上将 etcd 纳入监控。核心创建 Service/Endpoints Secret ServiceMonitor。1. 查看 etcd 证书与端口在 k8s master 节点操作。bash# 查看 etcd 启动参数中的证书路径 grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml输出示例text- --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.keybash# 直接测试 etcd metrics 接口 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.36.152:2379/metrics -k | tail -1 # 输出类似promhttp_metric_handler_requests_total{code503} 0bash# 查看 etcd 监听端口 ss -tunlp | grep etcd2. 创建 etcd Service 与 Endpointsetcd 是静态 Pod没有自动创建 Service需要手动创建 Endpoints 和 Service 供 Prometheus 发现。bashmkdir -p servicemonitor/etcd cd servicemonitor/etcd/ vim etcd-service.ymlyaml--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom # 标签供 Service 和 ServiceMonitor 匹配 name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.36.152 # etcd 节点 IP按实际修改 ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIPbashkubectl apply -f etcd-service.yml kubectl get svc etcd-prom -n kube-system输出示例textNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd-prom ClusterIP 10.100.206.20 none 2379/TCP 128mbash# 通过 Service 测试抓取 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://10.100.206.20:2379/metrics -k | tail -13. 创建 etcd 证书 SecretPrometheus 需要证书才能访问 etcd 的 https metrics。bashkubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key kubectl -n monitoring get secrets etcd-ssl输出示例textNAME TYPE DATA AGE etcd-ssl Opaque 3 130m4. Prometheus 挂载 etcd 证书bashcd /root/kube-prometheus/manifests/ vim prometheus-prometheus.yaml在spec末尾追加yamlsecrets: - etcd-ssl # 将 Secret 挂载到 Prometheus Podbash# 更新资源replace 会整体替换 kubectl replace -f prometheus-prometheus.yaml # 查看证书是否挂载成功 kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- \ ls /etc/prometheus/secrets/etcd-ssl5. 创建 ServiceMonitor让 Prometheus 自动发现并抓取 etcd 指标。bashcd - vim etcd-servicemonitor.yamlyaml--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd spec: jobLabel: k8s-etcd-cluster-apps endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: https # 使用 https 协议 tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true # 关闭证书校验测试环境方便 selector: matchLabels: app: etcd-prom # 匹配目标 Service 的标签 namespaceSelector: matchNames: - kube-system # 目标 Service 所在命名空间bashkubectl create -f etcd-servicemonitor.yaml kubectl get servicemonitor -n monitoring | grep etcd6. 查看监控数据与 Grafana 面板在 Prometheus UI 中查询 etcd 指标确认 Target 为 UP。Grafana 导入 etcd 监控面板texthttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/云原生应用监控etcd 集群监控笔记在 kube-prometheus 基础上将 etcd 纳入监控。核心创建 Service/Endpoints Secret ServiceMonitor。1. 查看 etcd 证书与端口在 k8s master 节点操作。bash# 查看 etcd 启动参数中的证书路径 grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml输出示例text- --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.keybash# 直接测试 etcd metrics 接口 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.36.152:2379/metrics -k | tail -1 # 输出类似promhttp_metric_handler_requests_total{code503} 0bash# 查看 etcd 监听端口 ss -tunlp | grep etcd2. 创建 etcd Service 与 Endpointsetcd 是静态 Pod没有自动创建 Service需要手动创建 Endpoints 和 Service 供 Prometheus 发现。bashmkdir -p servicemonitor/etcd cd servicemonitor/etcd/ vim etcd-service.ymlyaml--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom # 标签供 Service 和 ServiceMonitor 匹配 name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.36.152 # etcd 节点 IP按实际修改 ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIPbashkubectl apply -f etcd-service.yml kubectl get svc etcd-prom -n kube-system输出示例textNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd-prom ClusterIP 10.100.206.20 none 2379/TCP 128mbash# 通过 Service 测试抓取 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://10.100.206.20:2379/metrics -k | tail -13. 创建 etcd 证书 SecretPrometheus 需要证书才能访问 etcd 的 https metrics。bashkubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key kubectl -n monitoring get secrets etcd-ssl输出示例textNAME TYPE DATA AGE etcd-ssl Opaque 3 130m4. Prometheus 挂载 etcd 证书bashcd /root/kube-prometheus/manifests/ vim prometheus-prometheus.yaml在spec末尾追加yamlsecrets: - etcd-ssl # 将 Secret 挂载到 Prometheus Podbash# 更新资源replace 会整体替换 kubectl replace -f prometheus-prometheus.yaml # 查看证书是否挂载成功 kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- \ ls /etc/prometheus/secrets/etcd-ssl5. 创建 ServiceMonitor让 Prometheus 自动发现并抓取 etcd 指标。bashcd - vim etcd-servicemonitor.yamlyaml--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd spec: jobLabel: k8s-etcd-cluster-apps endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: https # 使用 https 协议 tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true # 关闭证书校验测试环境方便 selector: matchLabels: app: etcd-prom # 匹配目标 Service 的标签 namespaceSelector: matchNames: - kube-system # 目标 Service 所在命名空间bashkubectl create -f etcd-servicemonitor.yaml kubectl get servicemonitor -n monitoring | grep etcd6. 查看监控数据与 Grafana 面板在 Prometheus UI 中查询 etcd 指标确认 Target 为 UP。Grafana 导入 etcd 监控面板texthttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/此时没有集群查看不到状态云原生 kube-proxy 监控kube-proxy 默认只监听127.0.0.1:10249Prometheus 无法直接抓取。需要修改监听地址为0.0.0.0:10249再通过 Service ServiceMonitor 纳入监控。1. 查看默认监听与指标# 查看 kube-proxy 监听端口 ss -tunlp | grep kube-proxy # 输出tcp LISTEN 0 4096 127.0.0.1:10249 ... # 本机测试指标接口 curl 127.0.0.1:10249/metrics2. 修改 kube-proxy 监听地址kubectl edit configmap kube-proxy -n kube-system找到metricsBindAddress字段修改为metricsBindAddress: 0.0.0.0:10249 # 改为监听所有网卡允许外部抓取3. 重启 kube-proxy Pod 使配置生效kube-proxy 通常以 DaemonSet 部署删除 Pod 后会自动重建。# 删除所有 kube-proxy Pod按标签更简洁 kubectl -n kube-system delete pod -l k8s-appkube-proxy # 或者使用原文的 awk 方式 kubectl -n kube-system get pod | grep kube-proxy | \ awk {system(kubectl -n kube-system delete pod $1)}4. 创建 kube-proxy Service# kube-proxy-service.yaml apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-proxy # 标签供 ServiceMonitor 匹配 name: kube-proxy namespace: kube-system spec: selector: k8s-app: kube-proxy # 选中 kube-proxy Pod type: ClusterIP ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 10249 targetPort: 10249 protocol: TCPkubectl create -f kube-proxy-service.yaml kubectl get svc -n kube-system | grep kube-proxy输出示例kube-proxy ClusterIP 10.96.209.4 none 10249/TCP 22m# 通过 Service 测试抓取 curl 10.96.209.4:10249/metrics5. 创建 ServiceMonitor# kube-proxy-servicemonitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-proxy namespace: monitoring labels: app: kube-proxy spec: jobLabel: kube-proxy endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: http # kube-proxy 指标为 http selector: matchLabels: k8s-app: kube-proxy # 匹配 Service 的标签 namespaceSelector: matchNames: - kube-system # Service 所在命名空间kubectl create -f kube-proxy-servicemonitor.yaml kubectl get servicemonitor kube-proxy -n monitoring6. Prometheus 验证打开 Prometheus Web UI →Status → Targets查看kube-proxy是否出现且状态为UP。也可在查询框输入kube_proxy_*指标验证数据。grafana监控云原生 Calico 网络插件监控Calico 的核心组件是Felix负责路由表、ACL 规则及网络健康数据。监控 Calico 本质上就是监控 Felix。Calico 默认未开启 metrics 端口需手动开启并通过PodMonitor纳入 Prometheus。1. 查看 calico-node Podcalico-node 以 DaemonSet 部署每个节点一个 Pod。kubectl get po -n kube-system | grep calico-node输出示例calico-node-97mdd 1/1 Running 0 4m28s calico-node-l2b4b 1/1 Running 0 4m44s calico-node-q6rwq 1/1 Running 0 4m58s2. 开启 metrics 监听端口编辑 DaemonSetkubectl edit ds calico-node -n kube-system在containers的env中添加环境变量开启 Felix 的 metricsspec: containers: - env: - name: FELIX_PROMETHEUSMETRICSENABLED value: True - name: FELIX_PROMETHEUSMETRICSPORT value: 9091在ports中暴露端口ports: - containerPort: 9091 name: http-metrics # 端口名称PodMonitor 会引用 protocol: TCP注意 YAML 缩进保存后 calico-node Pod 会自动重建。3. 创建 PodMonitorcalico-node 是 Pod 而非 Service因此使用PodMonitor。yaml # prometheus-podMonitorCalico.yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: labels: k8s-app: calico-node name: calico-node namespace: monitoring spec: podMetricsEndpoints: - interval: 15s # 抓取间隔 path: /metrics # 指标路径 port: http-metrics # 对应容器端口名称 namespaceSelector: matchNames: - kube-system # calico-node 所在命名空间 selector: matchLabels: k8s-app: calico-node # 匹配 calico-node Pod 标签应用并查看kubectl apply -f prometheus-podMonitorCalico.yaml kubectl get podmonitor -n monitoring输出示例NAME AGE calico-node 8s4. Prometheus 验证打开 Prometheus Web UI →Status → Targets查看calico-node是否出现且状态为UP。也可查询felix_*指标确认数据。5. Grafana 仪表盘导入 Felix Dashboardtexthttps://grafana.com/grafana/dashboards/12175-felix-dashboard-calico/云原生 Nginx-Ingress 监控Nginx-Ingress 通过 Helm 部署时默认不会开启metrics 采集。需要修改 values 开启 metrics、Service 和 ServiceMonitor并处理 Helm 自动创建的 ServiceMonitor 标签不准确的问题。若 Prometheus 抓取失败还需检查日志并配置 RBAC 授权。1. 开启 metrics 采集修改 Helm values 文件开启 metrics 相关配置global: image: registry: reg.westos.org controller: image: image: ingress-nginx/controller tag: v1.15.1 digest: digestChroot: ingressClassResource: name: nginx default: true service: type: LoadBalancer admissionWebhooks: patch: image: image: ingress-nginx/kube-webhook-certgen tag: v1.6.9 digest: metrics: enabled: true service: enabled: true labels: app: ingress-nginx-metrics serviceMonitor: enabled: true namespace: monitoring defaultBackend: enabled: true name: defaultbackend image: registry: reg.westos.org image: ingress-nginx/defaultbackend-amd64 tag: 1.5说明metrics.enabled: true让 controller 暴露/metrics接口默认端口 10254。metrics.service.enabled: true创建ingress-nginx-controller-metricsService。metrics.serviceMonitor.enabled: true让 Helm 自动生成 ServiceMonitor但生成的 selector 通常不准确后续需要手动修改。2. 更新 Helm Releasehelm -n ingress-nginx upgrade ingress-nginx \ -f /resources/charts/values/ingress-nginx.yml \ /resources/charts/ingress-nginx-4.45.1.tgz说明使用新的 values 更新已有 releaseHelm 会创建/更新 metrics Service 和 ServiceMonitor。3. 验证 metrics Servicekubectl -n ingress-nginx get svc ingress-nginx-controller-metrics输出示例text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller-metrics ClusterIP 10.99.131.96 none 10254/TCP 15h测试拉取指标curl 10.99.131.96:10254/metrics说明如果能返回指标数据说明 metrics 端点正常。4. 修改 Helm 自动创建的 ServiceMonitorHelm 创建的 ServiceMonitor 的selector通常不匹配实际 Service 的标签需要手动修正。kubectl -n monitoring edit servicemonitors ingress-nginx-controller修改为yaml spec: endpoints: - interval: 30s port: metrics # 对应 Service 的端口名 scheme: http namespaceSelector: matchNames: - ingress-nginx # metrics Service 所在命名空间 selector: matchLabels: app: ingress-nginx-metrics # 匹配 metrics Service 的标签说明port: metrics必须与ingress-nginx-controller-metricsService 中定义的端口名一致。selector.matchLabels必须匹配 metrics Service 的标签即 values 中metrics.service.labels.app的值。namespaceSelector.matchNames指定 Service 所在的命名空间ingress-nginx。5. Prometheus 验证打开 Prometheus Web UI →Status → Targets查看ingress-nginx相关 Target 是否出现且状态为UP。也可查询nginx_ingress_controller_*指标确认数据。6. 排错与 RBAC 授权如果抓取不到数据查看 Prometheus 日志kubectl -n monitoring logs -f prometheus-k8s-0常见问题Prometheus 部署后安装的服务可能出现权限问题。因为 Prometheus 的 ServiceAccount 在monitoring命名空间而需要访问ingress-nginx命名空间的 endpoints、pods、services 等资源所以需要配置 RBAC 授权。创建rbac.ymlyaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: ingress-nginx rules: - apiGroups: [] resources: [endpoints, pods, services] verbs: [get, list, watch] - apiGroups: [discovery.k8s.io] resources: [endpointslices] verbs: [get, list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: ingress-nginx subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io应用授权kubectl apply -f rbac.yml说明Role 定义在ingress-nginx命名空间允许读取 endpoints、pods、services、endpointslices。RoleBinding 将monitoring命名空间中的prometheus-k8sServiceAccount 绑定到该 Role。这样 Prometheus 就能跨命名空间发现并抓取 ingress-nginx 的 metrics7. Grafana 仪表盘导入 Nginx Ingress Controller Dashboardhttps://grafana.com/grafana/dashboards/21336-nginx-ingress-controller/ 
长河君 · 2026/9/29 10:08:32 · 阅读 1.2k Kube-Prometheus部署
Kube-Prometheus 部署基于 kube-prometheus release-0.18包含镜像调整、副本数、持久化、Service 暴露等步骤。1. 下载安装包git clone -b release-0.18 https://github.com/prometheus-operator/kube-prometheus.git cd kube-prometheus/manifests/setup/ kubectl create -f . # 先创建 CRD 等基础资源 cd .. grep -r -i image: . # 查看所有镜像按需修改镜像地址编辑的镜像地址改成自己私有仓库再把导入的镜像tar包或者拉取的镜像打tag重新上传到私有仓库# 先登录私有仓库 docker login reg.westos.org -u admin -p westos # 1. alertmanager docker tag quay.io/prometheus/alertmanager:v0.33.0 reg.westos.org/prometheus/alertmanager:v0.33.0 docker push reg.westos.org/prometheus/alertmanager:v0.33.0 # 2. blackbox-exporter docker tag quay.io/prometheus/blackbox-exporter:v0.28.0 reg.westos.org/prometheus/blackbox-exporter:v0.28.0 docker push reg.westos.org/prometheus/blackbox-exporter:v0.28.0 # 3. kube-state-metrics docker tag registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.0 reg.westos.org/prometheus/kube-state-metrics:v2.19.0 docker push reg.westos.org/prometheus/kube-state-metrics:v2.19.0 # 4. node-exporter docker tag quay.io/prometheus/node-exporter:v1.11.1 reg.westos.org/prometheus/node-exporter:v1.11.1 docker push reg.westos.org/prometheus/node-exporter:v1.11.1 # 5. prometheus docker tag quay.io/prometheus/prometheus:v3.12.0 reg.westos.org/prometheus/prometheus:v3.12.0 docker push reg.westos.org/prometheus/prometheus:v3.12.0 # 6. prometheus-adapter docker tag registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0 reg.westos.org/prometheus/prometheus-adapter:v0.12.0 docker push reg.westos.org/prometheus/prometheus-adapter:v0.12.0 # 7. configmap-reload docker tag ghcr.io/jimmidyson/configmap-reload:v0.15.0 reg.westos.org/prometheus/configmap-reload:v0.15.0 docker push reg.westos.org/prometheus/configmap-reload:v0.15.0 # 8. grafana docker tag grafana/grafana:13.0.2 reg.westos.org/prometheus/grafana:13.0.2 docker push reg.westos.org/prometheus/grafana:13.0.2 # 9. kube-rbac-proxy docker tag quay.io/brancz/kube-rbac-proxy:v0.22.0 reg.westos.org/prometheus/kube-rbac-proxy:v0.22.0 docker push reg.westos.org/prometheus/kube-rbac-proxy:v0.22.0 # 10. prometheus-operator docker tag quay.io/prometheus-operator/prometheus-operator:v0.92.0 reg.westos.org/prometheus/prometheus-operator:v0.92.0 docker push reg.westos.org/prometheus/prometheus-operator:v0.92.02. 修改配置2.1 调整副本数# 修改 alertmanager 副本数为 1 vim alertmanager-alertmanager.yaml # 修改 replicas: 1 # 修改 prometheus 副本数为 1 vim prometheus-prometheus.yaml # 修改 replicas: 13. Grafana 数据持久化3.1 创建 PVCapiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pvc-nfs namespace: monitoring spec: accessModes: - ReadWriteMany # 多节点同时读写 resources: requests: storage: 10Gi # 存储大小 storageClassName: nfs-client # 使用 nfs-client StorageClasskubectl create -f grafana-pvc.yaml kubectl get pvc -n monitoring3.2 修改 Grafana Deploymentvim grafana-deployment.yaml修改 volumes 部分注意缩进yaml volumes: # - emptyDir: {} # name: grafana-storage - name: grafana-storage persistentVolumeClaim: claimName: grafana-pvc-nfs readOnly: false4. Prometheus 持久化配置vim prometheus-prometheus.yaml添加/修改yaml image: reg.westos.org/prometheus/prometheus:v3.5.0 storage: volumeClaimTemplate: spec: accessModes: [ ReadWriteOnce ] storageClassName: nfs-client resources: requests: storage: 30Gi5. 部署 Prometheus 服务kubectl create -f .6. 清除所有网络策略kubectl -n monitoring delete networkpolicies --all清除 NetworkPolicy 方便外部访问生产环境请按需保留。7. 暴露服务为 LoadBalancer7.1 Grafanakubectl get svc grafana -n monitoring kubectl edit svc grafana -n monitoring # 将 type 改为 LoadBalancer kubectl -n monitoring get svc grafana输出示例text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 10.99.135.255 192.168.36.101 3000:44532/TCP 159m访问http://192.168.36.101:3000用户名admin密码admin7.2 Prometheuskubectl get svc prometheus-k8s -n monitoring kubectl edit svc prometheus-k8s -n monitoring # 将 type 改为 LoadBalancer kubectl -n monitoring get svc prometheus-k8s输出示例text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus-k8s LoadBalancer 10.103.213.201 192.168.36.102 9090:33611/TCP,8080:45454/TCP 163m访问http://192.168.36.102:9090云原生应用监控etcd 集群监控在 kube-prometheus 基础上将 etcd 纳入监控。核心创建 Service/Endpoints Secret ServiceMonitor。1. 查看 etcd 证书与端口在 k8s master 节点操作。bash# 查看 etcd 启动参数中的证书路径 grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml输出示例text- --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.keybash# 直接测试 etcd metrics 接口 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.36.152:2379/metrics -k | tail -1 # 输出类似promhttp_metric_handler_requests_total{code503} 0bash# 查看 etcd 监听端口 ss -tunlp | grep etcd2. 创建 etcd Service 与 Endpointsetcd 是静态 Pod没有自动创建 Service需要手动创建 Endpoints 和 Service 供 Prometheus 发现。bashmkdir -p servicemonitor/etcd cd servicemonitor/etcd/ vim etcd-service.ymlyaml--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom # 标签供 Service 和 ServiceMonitor 匹配 name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.36.152 # etcd 节点 IP按实际修改 ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIPbashkubectl apply -f etcd-service.yml kubectl get svc etcd-prom -n kube-system输出示例textNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd-prom ClusterIP 10.100.206.20 none 2379/TCP 128mbash# 通过 Service 测试抓取 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://10.100.206.20:2379/metrics -k | tail -13. 创建 etcd 证书 SecretPrometheus 需要证书才能访问 etcd 的 https metrics。bashkubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key kubectl -n monitoring get secrets etcd-ssl输出示例textNAME TYPE DATA AGE etcd-ssl Opaque 3 130m4. Prometheus 挂载 etcd 证书bashcd /root/kube-prometheus/manifests/ vim prometheus-prometheus.yaml在spec末尾追加yamlsecrets: - etcd-ssl # 将 Secret 挂载到 Prometheus Podbash# 更新资源replace 会整体替换 kubectl replace -f prometheus-prometheus.yaml # 查看证书是否挂载成功 kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- \ ls /etc/prometheus/secrets/etcd-ssl5. 创建 ServiceMonitor让 Prometheus 自动发现并抓取 etcd 指标。bashcd - vim etcd-servicemonitor.yamlyaml--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd spec: jobLabel: k8s-etcd-cluster-apps endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: https # 使用 https 协议 tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true # 关闭证书校验测试环境方便 selector: matchLabels: app: etcd-prom # 匹配目标 Service 的标签 namespaceSelector: matchNames: - kube-system # 目标 Service 所在命名空间bashkubectl create -f etcd-servicemonitor.yaml kubectl get servicemonitor -n monitoring | grep etcd6. 查看监控数据与 Grafana 面板在 Prometheus UI 中查询 etcd 指标确认 Target 为 UP。Grafana 导入 etcd 监控面板texthttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/云原生应用监控etcd 集群监控笔记在 kube-prometheus 基础上将 etcd 纳入监控。核心创建 Service/Endpoints Secret ServiceMonitor。1. 查看 etcd 证书与端口在 k8s master 节点操作。bash# 查看 etcd 启动参数中的证书路径 grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml输出示例text- --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.keybash# 直接测试 etcd metrics 接口 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.36.152:2379/metrics -k | tail -1 # 输出类似promhttp_metric_handler_requests_total{code503} 0bash# 查看 etcd 监听端口 ss -tunlp | grep etcd2. 创建 etcd Service 与 Endpointsetcd 是静态 Pod没有自动创建 Service需要手动创建 Endpoints 和 Service 供 Prometheus 发现。bashmkdir -p servicemonitor/etcd cd servicemonitor/etcd/ vim etcd-service.ymlyaml--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom # 标签供 Service 和 ServiceMonitor 匹配 name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.36.152 # etcd 节点 IP按实际修改 ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIPbashkubectl apply -f etcd-service.yml kubectl get svc etcd-prom -n kube-system输出示例textNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd-prom ClusterIP 10.100.206.20 none 2379/TCP 128mbash# 通过 Service 测试抓取 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://10.100.206.20:2379/metrics -k | tail -13. 创建 etcd 证书 SecretPrometheus 需要证书才能访问 etcd 的 https metrics。bashkubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key kubectl -n monitoring get secrets etcd-ssl输出示例textNAME TYPE DATA AGE etcd-ssl Opaque 3 130m4. Prometheus 挂载 etcd 证书bashcd /root/kube-prometheus/manifests/ vim prometheus-prometheus.yaml在spec末尾追加yamlsecrets: - etcd-ssl # 将 Secret 挂载到 Prometheus Podbash# 更新资源replace 会整体替换 kubectl replace -f prometheus-prometheus.yaml # 查看证书是否挂载成功 kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- \ ls /etc/prometheus/secrets/etcd-ssl5. 创建 ServiceMonitor让 Prometheus 自动发现并抓取 etcd 指标。bashcd - vim etcd-servicemonitor.yamlyaml--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd spec: jobLabel: k8s-etcd-cluster-apps endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: https # 使用 https 协议 tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true # 关闭证书校验测试环境方便 selector: matchLabels: app: etcd-prom # 匹配目标 Service 的标签 namespaceSelector: matchNames: - kube-system # 目标 Service 所在命名空间bashkubectl create -f etcd-servicemonitor.yaml kubectl get servicemonitor -n monitoring | grep etcd6. 查看监控数据与 Grafana 面板在 Prometheus UI 中查询 etcd 指标确认 Target 为 UP。Grafana 导入 etcd 监控面板texthttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/云原生应用监控etcd 集群监控笔记在 kube-prometheus 基础上将 etcd 纳入监控。核心创建 Service/Endpoints Secret ServiceMonitor。1. 查看 etcd 证书与端口在 k8s master 节点操作。bash# 查看 etcd 启动参数中的证书路径 grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml输出示例text- --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.keybash# 直接测试 etcd metrics 接口 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.36.152:2379/metrics -k | tail -1 # 输出类似promhttp_metric_handler_requests_total{code503} 0bash# 查看 etcd 监听端口 ss -tunlp | grep etcd2. 创建 etcd Service 与 Endpointsetcd 是静态 Pod没有自动创建 Service需要手动创建 Endpoints 和 Service 供 Prometheus 发现。bashmkdir -p servicemonitor/etcd cd servicemonitor/etcd/ vim etcd-service.ymlyaml--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom # 标签供 Service 和 ServiceMonitor 匹配 name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.36.152 # etcd 节点 IP按实际修改 ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIPbashkubectl apply -f etcd-service.yml kubectl get svc etcd-prom -n kube-system输出示例textNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd-prom ClusterIP 10.100.206.20 none 2379/TCP 128mbash# 通过 Service 测试抓取 curl -s --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://10.100.206.20:2379/metrics -k | tail -13. 创建 etcd 证书 SecretPrometheus 需要证书才能访问 etcd 的 https metrics。bashkubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key kubectl -n monitoring get secrets etcd-ssl输出示例textNAME TYPE DATA AGE etcd-ssl Opaque 3 130m4. Prometheus 挂载 etcd 证书bashcd /root/kube-prometheus/manifests/ vim prometheus-prometheus.yaml在spec末尾追加yamlsecrets: - etcd-ssl # 将 Secret 挂载到 Prometheus Podbash# 更新资源replace 会整体替换 kubectl replace -f prometheus-prometheus.yaml # 查看证书是否挂载成功 kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- \ ls /etc/prometheus/secrets/etcd-ssl5. 创建 ServiceMonitor让 Prometheus 自动发现并抓取 etcd 指标。bashcd - vim etcd-servicemonitor.yamlyaml--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd spec: jobLabel: k8s-etcd-cluster-apps endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: https # 使用 https 协议 tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true # 关闭证书校验测试环境方便 selector: matchLabels: app: etcd-prom # 匹配目标 Service 的标签 namespaceSelector: matchNames: - kube-system # 目标 Service 所在命名空间bashkubectl create -f etcd-servicemonitor.yaml kubectl get servicemonitor -n monitoring | grep etcd6. 查看监控数据与 Grafana 面板在 Prometheus UI 中查询 etcd 指标确认 Target 为 UP。Grafana 导入 etcd 监控面板texthttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/此时没有集群查看不到状态云原生 kube-proxy 监控kube-proxy 默认只监听127.0.0.1:10249Prometheus 无法直接抓取。需要修改监听地址为0.0.0.0:10249再通过 Service ServiceMonitor 纳入监控。1. 查看默认监听与指标# 查看 kube-proxy 监听端口 ss -tunlp | grep kube-proxy # 输出tcp LISTEN 0 4096 127.0.0.1:10249 ... # 本机测试指标接口 curl 127.0.0.1:10249/metrics2. 修改 kube-proxy 监听地址kubectl edit configmap kube-proxy -n kube-system找到metricsBindAddress字段修改为metricsBindAddress: 0.0.0.0:10249 # 改为监听所有网卡允许外部抓取3. 重启 kube-proxy Pod 使配置生效kube-proxy 通常以 DaemonSet 部署删除 Pod 后会自动重建。# 删除所有 kube-proxy Pod按标签更简洁 kubectl -n kube-system delete pod -l k8s-appkube-proxy # 或者使用原文的 awk 方式 kubectl -n kube-system get pod | grep kube-proxy | \ awk {system(kubectl -n kube-system delete pod $1)}4. 创建 kube-proxy Service# kube-proxy-service.yaml apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-proxy # 标签供 ServiceMonitor 匹配 name: kube-proxy namespace: kube-system spec: selector: k8s-app: kube-proxy # 选中 kube-proxy Pod type: ClusterIP ports: - name: https-metrics # 端口名称ServiceMonitor 会引用 port: 10249 targetPort: 10249 protocol: TCPkubectl create -f kube-proxy-service.yaml kubectl get svc -n kube-system | grep kube-proxy输出示例kube-proxy ClusterIP 10.96.209.4 none 10249/TCP 22m# 通过 Service 测试抓取 curl 10.96.209.4:10249/metrics5. 创建 ServiceMonitor# kube-proxy-servicemonitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-proxy namespace: monitoring labels: app: kube-proxy spec: jobLabel: kube-proxy endpoints: - interval: 30s # 抓取间隔 port: https-metrics # 对应 Service.spec.ports.name scheme: http # kube-proxy 指标为 http selector: matchLabels: k8s-app: kube-proxy # 匹配 Service 的标签 namespaceSelector: matchNames: - kube-system # Service 所在命名空间kubectl create -f kube-proxy-servicemonitor.yaml kubectl get servicemonitor kube-proxy -n monitoring6. Prometheus 验证打开 Prometheus Web UI →Status → Targets查看kube-proxy是否出现且状态为UP。也可在查询框输入kube_proxy_*指标验证数据。grafana监控云原生 Calico 网络插件监控Calico 的核心组件是Felix负责路由表、ACL 规则及网络健康数据。监控 Calico 本质上就是监控 Felix。Calico 默认未开启 metrics 端口需手动开启并通过PodMonitor纳入 Prometheus。1. 查看 calico-node Podcalico-node 以 DaemonSet 部署每个节点一个 Pod。kubectl get po -n kube-system | grep calico-node输出示例calico-node-97mdd 1/1 Running 0 4m28s calico-node-l2b4b 1/1 Running 0 4m44s calico-node-q6rwq 1/1 Running 0 4m58s2. 开启 metrics 监听端口编辑 DaemonSetkubectl edit ds calico-node -n kube-system在containers的env中添加环境变量开启 Felix 的 metricsspec: containers: - env: - name: FELIX_PROMETHEUSMETRICSENABLED value: True - name: FELIX_PROMETHEUSMETRICSPORT value: 9091在ports中暴露端口ports: - containerPort: 9091 name: http-metrics # 端口名称PodMonitor 会引用 protocol: TCP注意 YAML 缩进保存后 calico-node Pod 会自动重建。3. 创建 PodMonitorcalico-node 是 Pod 而非 Service因此使用PodMonitor。yaml # prometheus-podMonitorCalico.yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: labels: k8s-app: calico-node name: calico-node namespace: monitoring spec: podMetricsEndpoints: - interval: 15s # 抓取间隔 path: /metrics # 指标路径 port: http-metrics # 对应容器端口名称 namespaceSelector: matchNames: - kube-system # calico-node 所在命名空间 selector: matchLabels: k8s-app: calico-node # 匹配 calico-node Pod 标签应用并查看kubectl apply -f prometheus-podMonitorCalico.yaml kubectl get podmonitor -n monitoring输出示例NAME AGE calico-node 8s4. Prometheus 验证打开 Prometheus Web UI →Status → Targets查看calico-node是否出现且状态为UP。也可查询felix_*指标确认数据。5. Grafana 仪表盘导入 Felix Dashboardtexthttps://grafana.com/grafana/dashboards/12175-felix-dashboard-calico/云原生 Nginx-Ingress 监控Nginx-Ingress 通过 Helm 部署时默认不会开启metrics 采集。需要修改 values 开启 metrics、Service 和 ServiceMonitor并处理 Helm 自动创建的 ServiceMonitor 标签不准确的问题。若 Prometheus 抓取失败还需检查日志并配置 RBAC 授权。1. 开启 metrics 采集修改 Helm values 文件开启 metrics 相关配置global: image: registry: reg.westos.org controller: image: image: ingress-nginx/controller tag: v1.15.1 digest: digestChroot: ingressClassResource: name: nginx default: true service: type: LoadBalancer admissionWebhooks: patch: image: image: ingress-nginx/kube-webhook-certgen tag: v1.6.9 digest: metrics: enabled: true service: enabled: true labels: app: ingress-nginx-metrics serviceMonitor: enabled: true namespace: monitoring defaultBackend: enabled: true name: defaultbackend image: registry: reg.westos.org image: ingress-nginx/defaultbackend-amd64 tag: 1.5说明metrics.enabled: true让 controller 暴露/metrics接口默认端口 10254。metrics.service.enabled: true创建ingress-nginx-controller-metricsService。metrics.serviceMonitor.enabled: true让 Helm 自动生成 ServiceMonitor但生成的 selector 通常不准确后续需要手动修改。2. 更新 Helm Releasehelm -n ingress-nginx upgrade ingress-nginx \ -f /resources/charts/values/ingress-nginx.yml \ /resources/charts/ingress-nginx-4.45.1.tgz说明使用新的 values 更新已有 releaseHelm 会创建/更新 metrics Service 和 ServiceMonitor。3. 验证 metrics Servicekubectl -n ingress-nginx get svc ingress-nginx-controller-metrics输出示例text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller-metrics ClusterIP 10.99.131.96 none 10254/TCP 15h测试拉取指标curl 10.99.131.96:10254/metrics说明如果能返回指标数据说明 metrics 端点正常。4. 修改 Helm 自动创建的 ServiceMonitorHelm 创建的 ServiceMonitor 的selector通常不匹配实际 Service 的标签需要手动修正。kubectl -n monitoring edit servicemonitors ingress-nginx-controller修改为yaml spec: endpoints: - interval: 30s port: metrics # 对应 Service 的端口名 scheme: http namespaceSelector: matchNames: - ingress-nginx # metrics Service 所在命名空间 selector: matchLabels: app: ingress-nginx-metrics # 匹配 metrics Service 的标签说明port: metrics必须与ingress-nginx-controller-metricsService 中定义的端口名一致。selector.matchLabels必须匹配 metrics Service 的标签即 values 中metrics.service.labels.app的值。namespaceSelector.matchNames指定 Service 所在的命名空间ingress-nginx。5. Prometheus 验证打开 Prometheus Web UI →Status → Targets查看ingress-nginx相关 Target 是否出现且状态为UP。也可查询nginx_ingress_controller_*指标确认数据。6. 排错与 RBAC 授权如果抓取不到数据查看 Prometheus 日志kubectl -n monitoring logs -f prometheus-k8s-0常见问题Prometheus 部署后安装的服务可能出现权限问题。因为 Prometheus 的 ServiceAccount 在monitoring命名空间而需要访问ingress-nginx命名空间的 endpoints、pods、services 等资源所以需要配置 RBAC 授权。创建rbac.ymlyaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: ingress-nginx rules: - apiGroups: [] resources: [endpoints, pods, services] verbs: [get, list, watch] - apiGroups: [discovery.k8s.io] resources: [endpointslices] verbs: [get, list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: ingress-nginx subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io应用授权kubectl apply -f rbac.yml说明Role 定义在ingress-nginx命名空间允许读取 endpoints、pods、services、endpointslices。RoleBinding 将monitoring命名空间中的prometheus-k8sServiceAccount 绑定到该 Role。这样 Prometheus 就能跨命名空间发现并抓取 ingress-nginx 的 metrics7. Grafana 仪表盘导入 Nginx Ingress Controller Dashboardhttps://grafana.com/grafana/dashboards/21336-nginx-ingress-controller/