Kubeadm 部署 Kubernetes v1.35 集群(RHEL9 + Harbor 私有仓库 + cri‑dockerd + Flannel) 一、环境规划主机名IP配置角色habor192.168.168.200cpu1、mem 1Gharbor 私有镜像仓库master‑node192.168.168.100cpu4、mem4Gk8s 控制节点 masternode1192.168.168.10cpu2、mem 2Gk8s 工作节点 workernode2192.168.168.20cpu2、mem 2Gk8s 工作节点 worker操作系统RHEL 9.3二、构建 harbor 镜像仓库192.168.168.2002.1. 部署 docker2.1.1部署docker本地仓库[rootharbor ~]# cat /etc/yum.repos.d/docker.repo EOF [docker] name docker baseurl https://mirrors.aliyun.com/docker-ce/linux/rhel/9.8/x86_64/stable/ gpgcheck 0 EOF #删除podman [rootHarbor ~]# dnf remove podman -y2.1.2安装docker在harbor仓库节点[rootharbor ~]# dnf install docker-ce -y [rootharbor ~]# echo br_netfilter /etc/modules-load.d/docker_mod.conf [rootharbor ~]# vim /etc/sysctl.d/docker.conf net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 [rootharbor ~]# systemctl restart systemd-modules-load.service [rootharbor ~]# sysctl --system #修改 docker service开启 iptables 参数 [rootharbor ~]# vim /lib/systemd/system/docker.service ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestrue [rootharbor ~]# systemctl daemon-reload [rootharbor ~]# systemctl enable --now docker2.2.生成自签名 SSL 证书[rootharbor ~]# mkdir /data/certs -p [rootharbor ~]# openssl req -newkey rsa:4096 \ -nodes -sha256 -keyout /data/certs/timinglee.org.key \ -addext subjectAltName DNS:reg.timinglee.org \ -x509 -days 365 -out /data/certs/timinglee.org.crt #填写证书信息 Country Name (2 letter code) [XX]:CN State or Province Name (full name) []:Shannxi Locality Name (eg, city) [Default City]:Xian Organization Name (eg, company) [Default Company Ltd]:kubernetes Organizational Unit Name (eg, section) []:harbor Common Name (eg, your name or your servers hostname) []:reg.timinglee.org Email Address []:admintiminglee.org2.3编辑 harbor 配置文件并安装提前准备harbor包harbor-offline-installer-v2.5.4.tgz#上传harbor包解压 [rootharbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/ [rootharbor ~]# cd /opt/harbor/ [rootharbor harbor]# ls common.sh harbor.v2.5.4.tar.gz harbor.yml.tmpl install.sh LICENSE prepare [rootharbor harbor]# cp harbor.yml.tmpl harbor.yml [rootharbor harbor]# vim harbor.yml hostname: reg.timinglee.org certificate: /data/certs/timinglee.org.crt private_key: /data/certs/timinglee.org.key harbor_admin_password: lee #安装harbor并开启chartmuseum helm仓库 [rootharbor harbor]# ./install.sh --with-chartmuseum2.3.1编写启动脚本[rootharbor ~]# vim /lib/systemd/system/harbor.service [Unit] Descriptionharbor with Docker Compose Documentationhttps://reg.timinglee.com/compose/ Afterdocker.service network-online.target Requiresdocker.service [Service] Typeoneshot WorkingDirectory/opt/harbor ExecStart/usr/bin/docker compose up -d ExecStop/usr/bin/docker compose down RemainAfterExityes [Install] WantedBymulti-user.target[rootharbor ~]# systemctl enable --now harbor [rootharbor ~]# systemctl enable --now harbor [rootharbor harbor]# docker compose ps2.4启动并验证 harbor[rootharbor harbor]# mkdir /etc/docker/certs.d/reg.timinglee.org/ -p [rootharbor harbor]# cp /data/certs/timinglee.org.crt /etc/docker/certs.d/reg.timinglee.org/ca.crt #系统全局信任ca证书 [rootHarbor ~]# cp /data/certs/timinglee.org.crt /etc/pki/ca-trust/source/anchors/ca.crt [rootHarbor ~]# update-ca-trust extract#hosts域名解析#配置docker镜像加速器 [rootHarbor ~]# cat /etc/docker/daemon.json EOF { registry-mirrors: [https://reg.timinglee.org] } EOF [rootharbor harbor]# systemctl restart docker [rootharbor harbor]# docker compose up -d #登录harbor [rootharbor harbor]# docker login reg.timinglee.org -u admin Password: lee Login Succeeded #测试镜像上传下载 [rootHarbor ~]# docker load -i busybox-latest.tar.gz [rootHarbor ~]# docker push reg.timinglee.org/library/busybox:latest The push refers to repository [reg.timinglee.org/library/busybox] d51af96cf93e: Pushed latest: digest: sha256:40680ace50cfe34f2180f482e3e8ee0dc8f87bb9b752da3a3a0dcc4616e78933 size: 400 [rootHarbor ~]# docker pull busybox:latest latest: Pulling from library/busybox Digest: sha256:40680ace50cfe34f2180f482e3e8ee0dc8f87bb9b752da3a3a0dcc4616e78933 Status: Downloaded newer image for busybox:latest docker.io/library/busybox:latest三、构建 kubernetes 集群主机环境master、node1、node2 全部执行3.1. 所有主机基础配置3.1.1关闭 swapk8s 强制要求#systemctl disable --now swap.target #systemctl mask swap.target #sed /swap/s/^/#/g -i /etc/fstab3.1.2harbor 节点分发证书到集群节点#harbor执行推送证书 [rootharbor ~]# for i in 100 10 20 do scp /data/certs/timinglee.org.crt root192.168.168.$i:/etc/docker/certs.d/reg.timinglee.org/ca.crt done3.1.3所有集群主机执行证书信任#mkdir /etc/docker/certs.d/reg.timinglee.org/ -p #cp /etc/docker/certs.d/reg.timinglee.org/ca.crt /etc/pki/ca-trust/source/anchors/ca.crt #update-ca-trust extract3.1.4所有主机配置内核参数#echo br_netfilter /etc/modules-load.d/docker_mod.conf #systemctl restart systemd-modules-load.service #cat /etc/sysctl.d/docker.conf EOF net.bridge.bridge-nf-call-ip6tables1 net.bridge.bridge-nf-call-iptables1 net.ipv4.ip_forward 1 EOF3.1.5docker 加速器配置所有节点#cat /etc/docker/daemon.json EOF { registry-mirrors:[https://reg.timinglee.org] } EOF #systemctl enable --now docker #docker info 可以看到 Registry Mirrors: https://reg.timinglee.org/3.1.6hosts 主机解析全部节点#vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.168.100 master 192.168.168.10 node1 192.168.168.20 node2 192.168.168.200 reg.timinglee.org3.1.7环境验证测试拉取镜像#docker pull busybox:latest latest: Pulling from library/busybox d51af96cf93e: Pull complete Digest: sha256:40680ace50cfe34f2180f482e3e8ee0dc8f87bb9b752da3a3a0dcc4616e78933 Status: Downloaded newer image for busybox:latest docker.io/library/busybox:latest3.1.8kubernetes 软件源所有节点#vim /etc/yum.repos.d/kubernetes.repo [kubernetes] name kubernetes baseurl https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/ gpgcheck 0 #dnf list kubelet重启服务器后执行swapon -s无输出代表 swap 彻底关闭。四、kubernetes 集群部署4.1. 所有主机安装 cri‑dockerd二选一rpm 包或者二进制部署4.1.1rpm 包部署版本较老[rootmaster1 ~]# rpm -ivh cri-dockerd-*.rpm #vim /lib/systemd/system/cri-docker.service ExecStart/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugincni --pod-infra-container-imagereg.timinglee.org/k8s/pause:3.10.1 ll /var/run/cri-dockerd.sock4.1.2二进制部署推荐#下载二进制包 #wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.9/cri-dockerd-0.4.9.amd64.tgz #解压 #tar zxf cri-dockerd-0.4.9.amd64.tgz #cd cri-dockerd/ #ls cri-dockerd cri-docker.service cri-docker.socket #安装cri-docker命令到系统 #install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd #生成启动文件 #cp cri-docker.s* /lib/systemd/system #chmod x /lib/systemd/system/cri-docker.* #修改service配置 #vim /lib/systemd/system/cri-docker.service [Service] Typenotify ExecStart/usr/local/bin/cri-dockerd --network-plugincni --pod-infra-container-imagereg.timinglee.org/k8s/pause:3.10.1 --container-runtime-endpoint fd:// ExecReload/bin/kill -s HUP $MAINPID TimeoutSec0 RestartSec2 Restartalways #systemctl daemon-reload # #systemctl enable --now cri-docker.service #systemctl status cri-docker.service4.2安装 构建k8s集群所需软件4.2.1所有节点部署k8s软件仓库# cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] name kubernetes baseurl https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/ gpgcheck 0 EOF4.2.2master 节点#dnf install kubelet kubeadm kubectl -y #systemctl enable --now kubelet.service #命令补全 #echo source (kubectl completion bash) ~/.bashrc #echo source (kubeadm completion bash) ~/.bashrc #source ~/.bashrc4.2.3node节点#dnf install kubelet kubeadm -y #systemctl enable --now kubelet.service4.3.master 节点下载集群镜像4.3.1查看需要镜像列表#[rootmaster ~]# kubeadm config images list4.3.2下载镜像阿里云镜像仓库#[rootmaster ~]# kubeadm config images pull \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.35.10 \ --cri-socketunix:///var/run/cri-dockerd.sock4.3.3镜像打标签推送到本地 harbor 私有仓库#登录仓库 #[rootmaster ~]# docker login reg.timinglee.org -u admin #打标签 #docker images --format {{.Repository}}:{{.Tag}} | awk -F / /google/{system(docker tag $0 reg.timinglee.org/k8s/$3)} #推送harbor #docker images --format {{.Repository}}:{{.Tag}} | awk -F / /timinglee/{system(docker push $0)}4.4.master 初始化 k8s 集群4.4.1在master中完成集群初始化#在master中完成集群初始化 #[rootmaster ~]# kubeadm init --pod-network-cidr10.244.0.0/16 \ --image-repository reg.timinglee.org/k8s \ --kubernetes-version v1.35.7 \ --cri-socketunix:///var/run/cri-dockerd.sock #输出示例 join 命令 #每个人不一样其他主机加入本集群的凭证 kubeadm join 192.168.168.100:6443 --token tdwjoc.8d1yw3wl4r4tm6c4 \ --discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927 #如果忘记重新生成加入命令 kubeadm token create --print-join-command #集群重置初始化失败使用 kubeadm reset --cri-socketunix:///var/run/cri-dockerd.sock4.4.2配置 kubectl 环境变量#[rootmaster ~]# echo export KUBECONFIG/etc/kubernetes/admin.conf ~/.bash_profile #[rootmaster ~]# source ~/.bash_profile #[rootmaster ~]# kubectl get nodes #此时节点状态 NotReady缺少网络插件4.4.3node 节点执行 join 加入集群#kubeadm join 192.168.168.100:6443 --token jl4ztx.cax3iysvu7onsh5s \ --discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927 \ --cri-socketunix:///var/run/cri-dockerd.sock #master 查看节点全部 NotReady #[rootmaster ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master NotReady control-plane 5m16s v1.35.3 node1 NotReady none 66s v1.35.3 node2 NotReady none 58s v1.35.34.5. 部署 flannel 网络插件4.5.1master 操作准备 flannel 离线包kube‑flannel‑v0.28.10.tar.gz[rootk8s-master ~]# tar zxf kube-flannel-v0.28.10.tar.gz [rootk8s-master ~]# cd flannel/ [rootk8s-master flannel]# docker load -i kube-flannel-v0.28.10.tar4.5.2flannel 镜像打标签推送到本地 harbor[rootmaster ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 [rootmaster ~]# docker push reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 [rootmaster ~]# docker tag ghcr.io/flannel-io/flannel:v0.2810 reg.timinglee.org/flannel-io/flannel:v0.28.10 [rootmaster ~]# docker push reg.timinglee.org/flannel-io/flannel:v0.28.104.5.3修改 yaml 内部镜像地址为 harbor 地址[rootk8s-master flannel]# grep -n image: kube-flannel.yml #要改内容所在行的位置 148: image: flannel-io/flannel:v0.28.10 175: image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3 186: image: flannel-io/flannel:v0.28.104.5.4应用 flannel 资源kubectl apply -f kube-flannel.yml4.5.5等待 pod 就绪验证节点全部 Ready[rootmaster ~]# kubectl get nodes