1. 为什么选择二进制部署K8s高可用集群在云原生技术栈中KubernetesK8s的部署方式主要有两种主流方案基于kubeadm的工具化部署和手动二进制部署。对于生产环境尤其是需要高度定制化的场景二进制部署方式具有独特优势全流程可控每个组件的启动参数、证书周期、日志级别都可精细调控排障直观组件间通过标准协议HTTP/GRPC通信可直接抓包分析升级灵活可逐个组件灰度升级避免kubeadm的全集群滚动重启资源占用低不依赖容器运行时部署控制平面内存开销减少约30%但硬币的另一面是复杂度陡增。以高可用架构为例需要手动处理证书轮换与聚合API配置多Master节点的etcd集群健康检查负载均衡器与健康检查策略的深度调优2. 基础设施准备与拓扑设计2.1 服务器规格建议根据实测数据推荐以下最小配置角色CPU内存磁盘数量云服务器实例类型示例Master4核8GB100GB SSD3阿里云ecs.c6.xlargeWorker8核16GB200GB SSD2腾讯云S5.2XLARGE16LB2核4GB40GB2华为云s6.large.2注意Master节点务必分散在不同可用区AZ避免单机房故障导致集群瘫痪2.2 网络规划要点Pod CIDR建议使用非标准私有网段如10.244.0.0/16避免与云平台VPC冲突Service CIDR推荐172.18.0.0/16需确保不与Node IP段重叠端口开放清单方向协议端口范围用途Master↔NodeTCP6443API ServerMaster↔MasterTCP2379-2380etcd peer communicationNode↔NodeUDP8472Flannel VXLANLB→MasterTCP6443健康检查端口3. 核心组件部署实战3.1 证书体系构建使用cfssl工具链生成四层证书# 生成CA配置 cat ca-config.json EOF { signing: { default: { expiry: 87600h }, profiles: { kubernetes: { usages: [signing, key encipherment, server auth, client auth], expiry: 87600h } } } } EOF # 生成etcd证书其他证书类似 cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json \ -hostname127.0.0.1,${MASTER_IPS},${CLUSTER_VIP} -profilekubernetes \ etcd-csr.json | cfssljson -bare etcd关键技巧证书有效期建议设置为10年87600h避免频繁轮换必须包含所有Master节点IP和负载均衡器VIPSANsSubject Alternative Names需配置kubernetes.default.svc.cluster.local等内部域名3.2 高可用etcd集群部署etcd集群的稳定性直接影响K8s API的可用性配置示例# /etc/etcd/etcd.conf name: etcd1>ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.1:2379 \ --cacert/etc/etcd/ssl/ca.pem \ --cert/etc/etcd/ssl/etcd.pem \ --key/etc/etcd/ssl/etcd-key.pem \ endpoint health常见问题处理节点间时钟偏差需小于1秒部署chronyd同步WAL日志堆积导致启动失败时清理/var/lib/etcd/member/wal/后重启内存不足时添加--auto-compaction-retention1h参数4. 负载均衡器深度配置4.1 云厂商LB选型对比云平台推荐LB类型优势注意事项阿里云应用型CLB支持加权轮询需手动配置TCP健康检查腾讯云传统型CLB免费提供带宽需单独购买AWSNLB保留源IP价格较高华为云共享型ELB跨AZ容灾最大连接数限制4.2 自建HAProxy方案对于无法使用云LB的场景推荐以下配置frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-masters backend k8s-masters mode tcp balance roundrobin option tcp-check server master1 192.168.1.1:6443 check fall 3 rise 2 server master2 192.168.1.2:6443 check fall 3 rise 2 server master3 192.168.1.3:6443 check fall 3 rise 2关键参数说明tcp-check发送SYN包检测端口存活比HTTP检查更高效fall 3连续3次失败才标记为DOWNrise 2连续2次成功恢复为UP配合Keepalived实现VIP漂移# /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } }5. 控制平面组件调优5.1 kube-apiserver关键参数--etcd-servershttps://192.168.1.1:2379,https://192.168.1.2:2379,https://192.168.1.3:2379 \ --bind-address0.0.0.0 \ --secure-port6443 \ --advertise-address192.168.1.100 \ --service-cluster-ip-range172.18.0.0/16 \ --enable-admission-pluginsNodeRestriction,NamespaceLifecycle \ --audit-log-path/var/log/k8s/audit.log \ --audit-log-maxage30 \ --request-timeout300s \ --max-mutating-requests-inflight600 \ --max-requests-inflight1200性能优化建议大型集群100节点需增加--max-requests-inflight启用--profilingtrue便于性能分析审计日志建议输出到单独磁盘5.2 高可用kube-controller-manager通过--leader-electtrue实现主备切换需特别注意--cluster-signing-cert-file/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file/etc/kubernetes/ssl/ca-key.pem \ --root-ca-file/etc/kubernetes/ssl/ca.pem \ --service-account-private-key-file/etc/kubernetes/ssl/sa.key \ --kubeconfig/etc/kubernetes/kcm.kubeconfig \ --use-service-account-credentialstrue \ --controllers*,bootstrapsigner,tokencleaner \ --node-monitor-period5s \ --node-monitor-grace-period40s重要所有Master节点必须使用相同的--cluster-cidr参数否则会导致路由冲突6. 工作节点接入与验证6.1 kubelet关键配置--bootstrap-kubeconfig/etc/kubernetes/bootstrap.kubeconfig \ --kubeconfig/etc/kubernetes/kubelet.kubeconfig \ --config/etc/kubernetes/kubelet-conf.yml \ --network-plugincni \ --cni-conf-dir/etc/cni/net.d \ --cni-bin-dir/opt/cni/bin \ --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.6 \ --rotate-certificatestrue \ --node-status-update-frequency10s其中kubelet-conf.yml需包含apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration evictionHard: memory.available: 500Mi nodefs.available: 10% imagefs.available: 15% maxPods: 150 failSwapOn: false containerLogMaxSize: 50Mi containerLogMaxFiles: 36.2 集群验收测试基础功能验证kubectl get cs # 检查组件健康状态 kubectl create deployment nginx --imagenginx:alpine kubectl expose deployment nginx --port80 curl $(kubectl get svc nginx -o jsonpath{.spec.clusterIP})高可用模拟测试随机停止一台Master节点的kube-apiserver观察kubectl命令是否仍可执行应有1-2秒短暂抖动通过VIP访问集群日志验证请求是否自动切换到健康节点压力测试建议kubectl apply -f https://k8s.io/examples/application/php-apache.yaml kubectl run -i --tty load-generator --rm --imagebusybox --restartNever \ -- /bin/sh -c while sleep 0.01; do wget -q -O- http://php-apache; done7. 生产环境加固建议7.1 网络策略配置默认情况下Pod间网络是全通的建议按需限制apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny spec: podSelector: {} policyTypes: - Ingress - Egress7.2 证书自动轮换通过kubelet配置开启自动轮换# /var/lib/kubelet/config.yaml rotateCertificates: true serverTLSBootstrap: true查看证书过期时间openssl x509 -noout -dates -in /var/lib/kubelet/pki/kubelet-client-current.pem7.3 关键监控指标Prometheus应监控的核心指标apiserver请求延迟apiserver_request_duration_seconds_bucketetcd写入性能etcd_disk_wal_fsync_duration_seconds_bucket节点内存压力node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes配置示例- alert: HighAPIServerLatency expr: histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le)) 1 for: 10m labels: severity: critical annotations: summary: API server latency high (instance {{ $labels.instance }}) description: API server 99th percentile latency is {{ $value }} seconds在实施过程中我发现二进制部署虽然初期投入较大但后期维护成本反而低于kubeadm方案。特别是在证书管理方面手动生成的CA体系比kubeadm自动生成的更易于维护。一个实用技巧是为每个组件创建单独的kubeconfig文件这样在权限回收时只需替换单个文件而无需全集群滚动重启。 
长河君 · 2026/10/29 12:48:19 · 阅读 1.2k 二进制部署K8s高可用集群实战指南
1. 为什么选择二进制部署K8s高可用集群在云原生技术栈中KubernetesK8s的部署方式主要有两种主流方案基于kubeadm的工具化部署和手动二进制部署。对于生产环境尤其是需要高度定制化的场景二进制部署方式具有独特优势全流程可控每个组件的启动参数、证书周期、日志级别都可精细调控排障直观组件间通过标准协议HTTP/GRPC通信可直接抓包分析升级灵活可逐个组件灰度升级避免kubeadm的全集群滚动重启资源占用低不依赖容器运行时部署控制平面内存开销减少约30%但硬币的另一面是复杂度陡增。以高可用架构为例需要手动处理证书轮换与聚合API配置多Master节点的etcd集群健康检查负载均衡器与健康检查策略的深度调优2. 基础设施准备与拓扑设计2.1 服务器规格建议根据实测数据推荐以下最小配置角色CPU内存磁盘数量云服务器实例类型示例Master4核8GB100GB SSD3阿里云ecs.c6.xlargeWorker8核16GB200GB SSD2腾讯云S5.2XLARGE16LB2核4GB40GB2华为云s6.large.2注意Master节点务必分散在不同可用区AZ避免单机房故障导致集群瘫痪2.2 网络规划要点Pod CIDR建议使用非标准私有网段如10.244.0.0/16避免与云平台VPC冲突Service CIDR推荐172.18.0.0/16需确保不与Node IP段重叠端口开放清单方向协议端口范围用途Master↔NodeTCP6443API ServerMaster↔MasterTCP2379-2380etcd peer communicationNode↔NodeUDP8472Flannel VXLANLB→MasterTCP6443健康检查端口3. 核心组件部署实战3.1 证书体系构建使用cfssl工具链生成四层证书# 生成CA配置 cat ca-config.json EOF { signing: { default: { expiry: 87600h }, profiles: { kubernetes: { usages: [signing, key encipherment, server auth, client auth], expiry: 87600h } } } } EOF # 生成etcd证书其他证书类似 cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json \ -hostname127.0.0.1,${MASTER_IPS},${CLUSTER_VIP} -profilekubernetes \ etcd-csr.json | cfssljson -bare etcd关键技巧证书有效期建议设置为10年87600h避免频繁轮换必须包含所有Master节点IP和负载均衡器VIPSANsSubject Alternative Names需配置kubernetes.default.svc.cluster.local等内部域名3.2 高可用etcd集群部署etcd集群的稳定性直接影响K8s API的可用性配置示例# /etc/etcd/etcd.conf name: etcd1>ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.1:2379 \ --cacert/etc/etcd/ssl/ca.pem \ --cert/etc/etcd/ssl/etcd.pem \ --key/etc/etcd/ssl/etcd-key.pem \ endpoint health常见问题处理节点间时钟偏差需小于1秒部署chronyd同步WAL日志堆积导致启动失败时清理/var/lib/etcd/member/wal/后重启内存不足时添加--auto-compaction-retention1h参数4. 负载均衡器深度配置4.1 云厂商LB选型对比云平台推荐LB类型优势注意事项阿里云应用型CLB支持加权轮询需手动配置TCP健康检查腾讯云传统型CLB免费提供带宽需单独购买AWSNLB保留源IP价格较高华为云共享型ELB跨AZ容灾最大连接数限制4.2 自建HAProxy方案对于无法使用云LB的场景推荐以下配置frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-masters backend k8s-masters mode tcp balance roundrobin option tcp-check server master1 192.168.1.1:6443 check fall 3 rise 2 server master2 192.168.1.2:6443 check fall 3 rise 2 server master3 192.168.1.3:6443 check fall 3 rise 2关键参数说明tcp-check发送SYN包检测端口存活比HTTP检查更高效fall 3连续3次失败才标记为DOWNrise 2连续2次成功恢复为UP配合Keepalived实现VIP漂移# /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } }5. 控制平面组件调优5.1 kube-apiserver关键参数--etcd-servershttps://192.168.1.1:2379,https://192.168.1.2:2379,https://192.168.1.3:2379 \ --bind-address0.0.0.0 \ --secure-port6443 \ --advertise-address192.168.1.100 \ --service-cluster-ip-range172.18.0.0/16 \ --enable-admission-pluginsNodeRestriction,NamespaceLifecycle \ --audit-log-path/var/log/k8s/audit.log \ --audit-log-maxage30 \ --request-timeout300s \ --max-mutating-requests-inflight600 \ --max-requests-inflight1200性能优化建议大型集群100节点需增加--max-requests-inflight启用--profilingtrue便于性能分析审计日志建议输出到单独磁盘5.2 高可用kube-controller-manager通过--leader-electtrue实现主备切换需特别注意--cluster-signing-cert-file/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file/etc/kubernetes/ssl/ca-key.pem \ --root-ca-file/etc/kubernetes/ssl/ca.pem \ --service-account-private-key-file/etc/kubernetes/ssl/sa.key \ --kubeconfig/etc/kubernetes/kcm.kubeconfig \ --use-service-account-credentialstrue \ --controllers*,bootstrapsigner,tokencleaner \ --node-monitor-period5s \ --node-monitor-grace-period40s重要所有Master节点必须使用相同的--cluster-cidr参数否则会导致路由冲突6. 工作节点接入与验证6.1 kubelet关键配置--bootstrap-kubeconfig/etc/kubernetes/bootstrap.kubeconfig \ --kubeconfig/etc/kubernetes/kubelet.kubeconfig \ --config/etc/kubernetes/kubelet-conf.yml \ --network-plugincni \ --cni-conf-dir/etc/cni/net.d \ --cni-bin-dir/opt/cni/bin \ --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.6 \ --rotate-certificatestrue \ --node-status-update-frequency10s其中kubelet-conf.yml需包含apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration evictionHard: memory.available: 500Mi nodefs.available: 10% imagefs.available: 15% maxPods: 150 failSwapOn: false containerLogMaxSize: 50Mi containerLogMaxFiles: 36.2 集群验收测试基础功能验证kubectl get cs # 检查组件健康状态 kubectl create deployment nginx --imagenginx:alpine kubectl expose deployment nginx --port80 curl $(kubectl get svc nginx -o jsonpath{.spec.clusterIP})高可用模拟测试随机停止一台Master节点的kube-apiserver观察kubectl命令是否仍可执行应有1-2秒短暂抖动通过VIP访问集群日志验证请求是否自动切换到健康节点压力测试建议kubectl apply -f https://k8s.io/examples/application/php-apache.yaml kubectl run -i --tty load-generator --rm --imagebusybox --restartNever \ -- /bin/sh -c while sleep 0.01; do wget -q -O- http://php-apache; done7. 生产环境加固建议7.1 网络策略配置默认情况下Pod间网络是全通的建议按需限制apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny spec: podSelector: {} policyTypes: - Ingress - Egress7.2 证书自动轮换通过kubelet配置开启自动轮换# /var/lib/kubelet/config.yaml rotateCertificates: true serverTLSBootstrap: true查看证书过期时间openssl x509 -noout -dates -in /var/lib/kubelet/pki/kubelet-client-current.pem7.3 关键监控指标Prometheus应监控的核心指标apiserver请求延迟apiserver_request_duration_seconds_bucketetcd写入性能etcd_disk_wal_fsync_duration_seconds_bucket节点内存压力node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes配置示例- alert: HighAPIServerLatency expr: histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le)) 1 for: 10m labels: severity: critical annotations: summary: API server latency high (instance {{ $labels.instance }}) description: API server 99th percentile latency is {{ $value }} seconds在实施过程中我发现二进制部署虽然初期投入较大但后期维护成本反而低于kubeadm方案。特别是在证书管理方面手动生成的CA体系比kubeadm自动生成的更易于维护。一个实用技巧是为每个组件创建单独的kubeconfig文件这样在权限回收时只需替换单个文件而无需全集群滚动重启。