1. 为什么要在Kubernetes上运行虚拟机当第一次听说KubeVirt这个项目时我和大多数Kubernetes用户一样感到困惑我们已经有了容器这种轻量级的虚拟化方案为什么还要在Kubernetes上运行传统的虚拟机这个问题困扰了我很久直到在实际生产环境中遇到了几个典型场景场景一遗留系统迁移我们有一个运行了十年的财务系统基于CentOS 6开发依赖特定的内核模块和硬件驱动。直接容器化需要重写大量代码而使用KubeVirt可以原封不动地将整个VM镜像迁移到Kubernetes集群仅用一周就完成了原本预估三个月的迁移工作。场景二GPU虚拟化需求AI团队需要为每个研究员分配带GPU的隔离环境。传统方案是给每人分配物理机利用率不到30%。通过KubeVirt的vGPU支持我们实现了单块A100显卡最多分割给8个VM使用动态调整各VM的显存配额从1GB到16GB通过Kubernetes的ResourceQuota控制总GPU使用量场景三安全隔离要求某政府项目要求不同安全等级的业务必须运行在硬件隔离的环境中。虽然容器有namespace隔离但某些审计标准仍只认可VM级别的隔离。KubeVirt配合Intel SGX技术让我们既能满足合规要求又能享受Kubernetes的编排能力。关键认知转折KubeVirt不是要用VM替代容器而是为特定场景提供逃生舱。根据我们的经验适合迁移到KubeVirt的工作负载通常具有以下特征强依赖特定内核版本或内核模块需要独占硬件资源如GPU、FPGA符合行业合规要求的隔离级别已有成熟的VM镜像且重构成本高2. KubeVirt架构深度解析2.1 核心组件协作关系第一次部署KubeVirt时我被它复杂的组件交互搞晕了。直到画出下面这张架构图才真正理解各组件如何协同工作----------------------- | KubeVirt Control | | Plane Components | ---------------------- | v ---------------------- ----------------------- | Virt-Controller |---| Kubernetes API | ---------------------- ----------------------- | v ---------------------- ----------------------- | Virt-Handler |---| Kubelet | ---------------------- ----------------------- | v ---------------------- ----------------------- | Virt-Launcher |---| Container Runtime | ---------------------- ----------------------- | v ---------------------- | Libvirtd/QEMU | -----------------------组件职责详解Virt-Controller相当于Kubernetes的Controller Manager持续监听VM、VMI等CRD对象的变化。我们曾遇到一个故障批量创建100个VM时有15个卡在Pending状态。最终发现是Virt-Controller的默认并发处理数--concurrent-vm-syncs5导致的调整到20后问题解决。Virt-Handler每个节点上运行的DaemonSet负责本节点VM的生命周期管理。关键细节通过unix socket与virt-launcher通信默认检查节点虚拟化能力的间隔是5分钟可通过--host-check-update-frequency调整会为每个VM创建对应的podVirt-Launcher实际运行VM的瘦容器内部封装了libvirtd和QEMU。我们做过性能测试单个virt-launcher容器内存开销约50MB启动一个空载CentOS VM需要约2秒支持热迁移的VM需要额外300MB内存用于脏页跟踪2.2 网络与存储方案选型网络方案对比实践方案类型实现方式优点缺点适用场景默认Pod网络通过multus分配Pod网络零配置性能差实测吞吐1Gbps测试环境SR-IOV通过SRIOV Device Plugin接近物理网卡性能需要硬件支持金融交易系统Macvtap创建macvtap接口性能损失5%需要特权容器通用生产环境DPDK用户态网络栈超高吞吐实测200Gbps配置复杂占用大量内存电信级NFV存储方案实战经验我们在生产环境主要使用两种模式PVCCSI动态供给适合有状态工作负载关键配置domain: devices: disks: - disk: bus: virtio name: pvcdisk volumes: - name: pvcdisk persistentVolumeClaim: claimName: my-vm-pvc实测性能Ceph RBD: 4K随机读写约8000 IOPSLocal PV: 4K随机读写可达50,000 IOPSContainerDisk只读CloudInit配置注入适合无状态工作负载启动速度比PVC快30%volumes: - containerDisk: image: registry/vm-images/centos7:latest name: rootdisk - cloudInitNoCloud: userData: |- #cloud-config password: mypass chpasswd: { expire: False }3. 生产环境部署实战3.1 硬件准备与性能调优CPU虚拟化检测与开启很多同学在笔记本上测试时遇到此平台不支持虚拟化的amd-v/rvi错误解决方法# 检查虚拟化支持 grep -E (vmx|svm) /proc/cpuinfo # 对于Intel CPU cat /sys/module/kvm_intel/parameters/nested # 应该返回Y # 开启嵌套虚拟化适用于物理机 echo options kvm-intel nestedY /etc/modprobe.d/kvm-intel.conf内存大页配置对于内存敏感的数据库类VM必须配置大页# 查看当前大页配置 grep HugePages /proc/meminfo # 永久配置示例分配8GB大页 echo vm.nr_hugepages 4096 /etc/sysctl.conf echo vm.hugetlb_shm_group 0 /etc/sysctl.conf sysctl -p # 在VM中挂载大页 domain: memory: hugepages: pageSize: 2Mi guest: 8GiNUMA调优我们有一个MySQL VM在48核机器上性能异常最终通过NUMA绑定解决domain: cpu: cores: 8 sockets: 1 threads: 1 numa: guestMappingPassthrough: {} resources: requests: memory: 16Gi limits: memory: 16Gi overcommitGuestOverhead: true3.2 安装与配置全流程使用kubevirt-operator部署推荐# 安装KubeVirt Operator export VERSION$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases | grep tag_name | grep -v -- -rc | head -1 | awk -F: {print $2} | tr -d ,) kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml # 部署CR实例 cat EOF | kubectl apply -f - apiVersion: kubevirt.io/v1 kind: KubeVirt metadata: name: kubevirt namespace: kubevirt spec: certificateRotateStrategy: {} configuration: developerConfiguration: featureGates: - LiveMigration - CPUManager permittedHostDevices: pciHostDevices: - resourceName: nvidia.com/gpu selector: 10de:13f8 # Tesla T4 externalResourceProvider: true workloadUpdateStrategy: batchEvictionSize: 10 EOF # 等待所有pod就绪 kubectl -n kubevirt wait --forconditionAvailable --all deployments关键配置解析featureGates必须显式开启LiveMigration才能支持热迁移permittedHostDevices允许VM直通GPU等设备batchEvictionSize滚动更新时最大并行更新数3.3 创建第一个虚拟机使用virtctl快速创建# 下载virtctl客户端 VERSION$(kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -ojsonpath{.status.observedKubeVirtVersion}) ARCH$(uname -s | tr A-Z a-z)-$(uname -m | sed s/x86_64/amd64/) || windows-amd64.exe curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-${ARCH} chmod x virtctl # 创建CentOS7 VM ./virtctl create vm --namemy-vm --imagequay.io/kubevirt/cirros-container-disk-demo --memory1Gi通过YAML声明式创建生产推荐apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: my-vm spec: running: true template: metadata: labels: kubevirt.io/domain: my-vm spec: domain: devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - name: default masquerade: {} resources: requests: memory: 2Gi networks: - name: default pod: {} volumes: - name: rootdisk containerDisk: image: quay.io/kubevirt/cirros-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: |- #cloud-config user: test password: test chpasswd: { expire: False }4. 高级特性与生产实践4.1 虚拟机热迁移实战我们在金融行业客户的生产环境实现了跨机房的VM热迁移关键步骤前提条件检查# 检查集群节点是否在同一个迁移域 kubectl get kubevirt -n kubevirt -ojsonpath{.items[0].status.migrations.configuration.clusterMigrations} # 检查网络带宽建议至少10Gbps kubectl virt migrate --vm-namemy-vm --target-nodenode02 --bandwidth10Gi迁移过程监控# 查看迁移进度 kubectl get vmi my-vm -o jsonpath{.status.migrationState} # 关键指标解释 # - completed: 已迁移数据百分比 # - remaining: 剩余数据量 # - duration: 已耗时性能优化参数spec: template: spec: evictionStrategy: LiveMigrate domain: devices: disks: - disk: cache: none # 对于IO密集型VM建议关闭缓存 cpu: model: host-passthrough resources: limits: cpu: 4 memory: 8Gi4.2 设备直通方案GPU直通配置示例domain: devices: gpus: - deviceName: nvidia.com/gpu name: gpu1 hostDevices: - deviceName: nvidia.com/gpu name: gpu2 tag: 10de:13f8 # 设备IDSR-IOV网卡直通domain: devices: interfaces: - name: sriov-net sriov: {} networks: - name: sriov-net multus: networkName: sriov-network直通设备的重要限制每个设备只能分配给一个VM设备驱动程序需兼容目标VM操作系统直通后主机无法使用该设备4.3 监控与日志方案Prometheus监控指标# kubevirt-prometheus-metrics.yaml apiVersion: v1 kind: Service metadata: name: kubevirt-prometheus-metrics namespace: kubevirt labels: prometheus.kubevirt.io: true spec: ports: - name: metrics port: 443 targetPort: 8443 selector: prometheus.kubevirt.io: true关键监控指标kubevirt_vmi_memory_available_bytesVM可用内存kubevirt_vmi_cpu_usage_secondsCPU使用时间kubevirt_vmi_network_traffic_bytes_total网络流量kubevirt_vmi_storage_iops_total磁盘IOPS日志收集技巧# 查看virt-launcher日志 kubectl logs -f $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) # 获取libvirtd调试日志 kubectl exec -it $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) -- \ virsh dumpxml my-vm | grep log -A5 # 持续日志收集方案 kubectl virt logs my-vm --since1h --follow5. 故障排查与性能优化5.1 常见问题排查指南问题一VM卡在Pending状态# 检查事件日志 kubectl describe vmi my-vm # 常见原因 # 1. 节点资源不足 - 检查kubectl describe node # 2. 没有满足条件的节点 - 检查nodeSelector和污点 # 3. PVC未就绪 - 检查kubectl get pvc问题二VM启动失败# 获取控制台输出 kubectl virt console my-vm # 检查QEMU日志 kubectl exec -it $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) -- \ cat /var/log/qemu.log # 常见错误 # - Failed to start domain: internal error: early end of file from monitor # 通常是镜像损坏重新导入镜像解决问题三网络连接异常# 进入virt-launcher调试网络 kubectl exec -it $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) -- bash # 检查tap设备 ip link show # 检查iptables规则masquerade模式 iptables -t nat -L -n -v5.2 性能优化实战CPU绑核配置domain: cpu: cores: 4 sockets: 1 threads: 1 dedicatedCpuPlacement: true isolateEmulatorThread: true resources: requests: cpu: 4 memory: 8Gi limits: cpu: 4 memory: 8Gi磁盘缓存策略对比测试缓存模式4K随机读(IOPS)顺序写(MB/s)适用场景none78,000520数据库类工作负载writeback65,000680通用场景writethru58,000450数据安全优先网络性能调优参数domain: devices: interfaces: - name: default model: virtio ports: - name: http port: 8080 bandwidth: ingress: 1000 egress: 1000 tune: queues: 4 # 多队列数建议等于vCPU数6. 安全加固实践6.1 最小权限原则实施ServiceAccount权限控制apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-user rules: - apiGroups: [kubevirt.io] resources: [virtualmachines, virtualmachineinstances] verbs: [get, list, create, delete] --- apiVersion: v1 kind: ServiceAccount metadata: name: vm-operator --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-user-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubevirt-user subjects: - kind: ServiceAccount name: vm-operatorPod安全策略apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: kubevirt spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - secret hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot seLinux: rule: RunAsAny supplementalGroups: rule: MustRunAs ranges: - min: 1 max: 65535 fsGroup: rule: MustRunAs ranges: - min: 1 max: 655356.2 镜像安全扫描使用Anchore扫描VM镜像# 安装anchore-cli pip install anchorecli # 扫描镜像 anchore-cli image add quay.io/kubevirt/cirros-container-disk-demo anchore-cli image wait quay.io/kubevirt/cirros-container-disk-demo anchore-cli image vuln quay.io/kubevirt/cirros-container-disk-demo all # 集成到CI/CD anchore-cli policy add ./security-policy.json anchore-cli evaluate check \ quay.io/kubevirt/cirros-container-disk-demo \ --detail关键安全策略禁止使用latest标签基础镜像仅来自受信任仓库定期更新镜像中的软件包启用SELinux或AppArmor6.3 网络隔离方案多租户网络隔离apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: tenant-a-net annotations: k8s.v1.cni.cncf.io/resourceName: macvlan.network/kubevirt spec: config: { cniVersion: 0.3.1, name: tenant-a-net, type: macvlan, master: eth0, mode: bridge, ipam: { type: whereabouts, range: 192.168.1.0/24, exclude: [192.168.1.1/32] } }网络策略示例apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vm-isolation spec: podSelector: matchLabels: kubevirt.io/domain: my-vm policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 80 egress: - to: - podSelector: matchLabels: role: database ports: - protocol: TCP port: 3306 
长河君 · 2026/10/28 3:47:47 · 阅读 1.2k KubeVirt实战：在Kubernetes上运行虚拟机的完整指南
1. 为什么要在Kubernetes上运行虚拟机当第一次听说KubeVirt这个项目时我和大多数Kubernetes用户一样感到困惑我们已经有了容器这种轻量级的虚拟化方案为什么还要在Kubernetes上运行传统的虚拟机这个问题困扰了我很久直到在实际生产环境中遇到了几个典型场景场景一遗留系统迁移我们有一个运行了十年的财务系统基于CentOS 6开发依赖特定的内核模块和硬件驱动。直接容器化需要重写大量代码而使用KubeVirt可以原封不动地将整个VM镜像迁移到Kubernetes集群仅用一周就完成了原本预估三个月的迁移工作。场景二GPU虚拟化需求AI团队需要为每个研究员分配带GPU的隔离环境。传统方案是给每人分配物理机利用率不到30%。通过KubeVirt的vGPU支持我们实现了单块A100显卡最多分割给8个VM使用动态调整各VM的显存配额从1GB到16GB通过Kubernetes的ResourceQuota控制总GPU使用量场景三安全隔离要求某政府项目要求不同安全等级的业务必须运行在硬件隔离的环境中。虽然容器有namespace隔离但某些审计标准仍只认可VM级别的隔离。KubeVirt配合Intel SGX技术让我们既能满足合规要求又能享受Kubernetes的编排能力。关键认知转折KubeVirt不是要用VM替代容器而是为特定场景提供逃生舱。根据我们的经验适合迁移到KubeVirt的工作负载通常具有以下特征强依赖特定内核版本或内核模块需要独占硬件资源如GPU、FPGA符合行业合规要求的隔离级别已有成熟的VM镜像且重构成本高2. KubeVirt架构深度解析2.1 核心组件协作关系第一次部署KubeVirt时我被它复杂的组件交互搞晕了。直到画出下面这张架构图才真正理解各组件如何协同工作----------------------- | KubeVirt Control | | Plane Components | ---------------------- | v ---------------------- ----------------------- | Virt-Controller |---| Kubernetes API | ---------------------- ----------------------- | v ---------------------- ----------------------- | Virt-Handler |---| Kubelet | ---------------------- ----------------------- | v ---------------------- ----------------------- | Virt-Launcher |---| Container Runtime | ---------------------- ----------------------- | v ---------------------- | Libvirtd/QEMU | -----------------------组件职责详解Virt-Controller相当于Kubernetes的Controller Manager持续监听VM、VMI等CRD对象的变化。我们曾遇到一个故障批量创建100个VM时有15个卡在Pending状态。最终发现是Virt-Controller的默认并发处理数--concurrent-vm-syncs5导致的调整到20后问题解决。Virt-Handler每个节点上运行的DaemonSet负责本节点VM的生命周期管理。关键细节通过unix socket与virt-launcher通信默认检查节点虚拟化能力的间隔是5分钟可通过--host-check-update-frequency调整会为每个VM创建对应的podVirt-Launcher实际运行VM的瘦容器内部封装了libvirtd和QEMU。我们做过性能测试单个virt-launcher容器内存开销约50MB启动一个空载CentOS VM需要约2秒支持热迁移的VM需要额外300MB内存用于脏页跟踪2.2 网络与存储方案选型网络方案对比实践方案类型实现方式优点缺点适用场景默认Pod网络通过multus分配Pod网络零配置性能差实测吞吐1Gbps测试环境SR-IOV通过SRIOV Device Plugin接近物理网卡性能需要硬件支持金融交易系统Macvtap创建macvtap接口性能损失5%需要特权容器通用生产环境DPDK用户态网络栈超高吞吐实测200Gbps配置复杂占用大量内存电信级NFV存储方案实战经验我们在生产环境主要使用两种模式PVCCSI动态供给适合有状态工作负载关键配置domain: devices: disks: - disk: bus: virtio name: pvcdisk volumes: - name: pvcdisk persistentVolumeClaim: claimName: my-vm-pvc实测性能Ceph RBD: 4K随机读写约8000 IOPSLocal PV: 4K随机读写可达50,000 IOPSContainerDisk只读CloudInit配置注入适合无状态工作负载启动速度比PVC快30%volumes: - containerDisk: image: registry/vm-images/centos7:latest name: rootdisk - cloudInitNoCloud: userData: |- #cloud-config password: mypass chpasswd: { expire: False }3. 生产环境部署实战3.1 硬件准备与性能调优CPU虚拟化检测与开启很多同学在笔记本上测试时遇到此平台不支持虚拟化的amd-v/rvi错误解决方法# 检查虚拟化支持 grep -E (vmx|svm) /proc/cpuinfo # 对于Intel CPU cat /sys/module/kvm_intel/parameters/nested # 应该返回Y # 开启嵌套虚拟化适用于物理机 echo options kvm-intel nestedY /etc/modprobe.d/kvm-intel.conf内存大页配置对于内存敏感的数据库类VM必须配置大页# 查看当前大页配置 grep HugePages /proc/meminfo # 永久配置示例分配8GB大页 echo vm.nr_hugepages 4096 /etc/sysctl.conf echo vm.hugetlb_shm_group 0 /etc/sysctl.conf sysctl -p # 在VM中挂载大页 domain: memory: hugepages: pageSize: 2Mi guest: 8GiNUMA调优我们有一个MySQL VM在48核机器上性能异常最终通过NUMA绑定解决domain: cpu: cores: 8 sockets: 1 threads: 1 numa: guestMappingPassthrough: {} resources: requests: memory: 16Gi limits: memory: 16Gi overcommitGuestOverhead: true3.2 安装与配置全流程使用kubevirt-operator部署推荐# 安装KubeVirt Operator export VERSION$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases | grep tag_name | grep -v -- -rc | head -1 | awk -F: {print $2} | tr -d ,) kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml # 部署CR实例 cat EOF | kubectl apply -f - apiVersion: kubevirt.io/v1 kind: KubeVirt metadata: name: kubevirt namespace: kubevirt spec: certificateRotateStrategy: {} configuration: developerConfiguration: featureGates: - LiveMigration - CPUManager permittedHostDevices: pciHostDevices: - resourceName: nvidia.com/gpu selector: 10de:13f8 # Tesla T4 externalResourceProvider: true workloadUpdateStrategy: batchEvictionSize: 10 EOF # 等待所有pod就绪 kubectl -n kubevirt wait --forconditionAvailable --all deployments关键配置解析featureGates必须显式开启LiveMigration才能支持热迁移permittedHostDevices允许VM直通GPU等设备batchEvictionSize滚动更新时最大并行更新数3.3 创建第一个虚拟机使用virtctl快速创建# 下载virtctl客户端 VERSION$(kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -ojsonpath{.status.observedKubeVirtVersion}) ARCH$(uname -s | tr A-Z a-z)-$(uname -m | sed s/x86_64/amd64/) || windows-amd64.exe curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-${ARCH} chmod x virtctl # 创建CentOS7 VM ./virtctl create vm --namemy-vm --imagequay.io/kubevirt/cirros-container-disk-demo --memory1Gi通过YAML声明式创建生产推荐apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: my-vm spec: running: true template: metadata: labels: kubevirt.io/domain: my-vm spec: domain: devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - name: default masquerade: {} resources: requests: memory: 2Gi networks: - name: default pod: {} volumes: - name: rootdisk containerDisk: image: quay.io/kubevirt/cirros-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: |- #cloud-config user: test password: test chpasswd: { expire: False }4. 高级特性与生产实践4.1 虚拟机热迁移实战我们在金融行业客户的生产环境实现了跨机房的VM热迁移关键步骤前提条件检查# 检查集群节点是否在同一个迁移域 kubectl get kubevirt -n kubevirt -ojsonpath{.items[0].status.migrations.configuration.clusterMigrations} # 检查网络带宽建议至少10Gbps kubectl virt migrate --vm-namemy-vm --target-nodenode02 --bandwidth10Gi迁移过程监控# 查看迁移进度 kubectl get vmi my-vm -o jsonpath{.status.migrationState} # 关键指标解释 # - completed: 已迁移数据百分比 # - remaining: 剩余数据量 # - duration: 已耗时性能优化参数spec: template: spec: evictionStrategy: LiveMigrate domain: devices: disks: - disk: cache: none # 对于IO密集型VM建议关闭缓存 cpu: model: host-passthrough resources: limits: cpu: 4 memory: 8Gi4.2 设备直通方案GPU直通配置示例domain: devices: gpus: - deviceName: nvidia.com/gpu name: gpu1 hostDevices: - deviceName: nvidia.com/gpu name: gpu2 tag: 10de:13f8 # 设备IDSR-IOV网卡直通domain: devices: interfaces: - name: sriov-net sriov: {} networks: - name: sriov-net multus: networkName: sriov-network直通设备的重要限制每个设备只能分配给一个VM设备驱动程序需兼容目标VM操作系统直通后主机无法使用该设备4.3 监控与日志方案Prometheus监控指标# kubevirt-prometheus-metrics.yaml apiVersion: v1 kind: Service metadata: name: kubevirt-prometheus-metrics namespace: kubevirt labels: prometheus.kubevirt.io: true spec: ports: - name: metrics port: 443 targetPort: 8443 selector: prometheus.kubevirt.io: true关键监控指标kubevirt_vmi_memory_available_bytesVM可用内存kubevirt_vmi_cpu_usage_secondsCPU使用时间kubevirt_vmi_network_traffic_bytes_total网络流量kubevirt_vmi_storage_iops_total磁盘IOPS日志收集技巧# 查看virt-launcher日志 kubectl logs -f $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) # 获取libvirtd调试日志 kubectl exec -it $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) -- \ virsh dumpxml my-vm | grep log -A5 # 持续日志收集方案 kubectl virt logs my-vm --since1h --follow5. 故障排查与性能优化5.1 常见问题排查指南问题一VM卡在Pending状态# 检查事件日志 kubectl describe vmi my-vm # 常见原因 # 1. 节点资源不足 - 检查kubectl describe node # 2. 没有满足条件的节点 - 检查nodeSelector和污点 # 3. PVC未就绪 - 检查kubectl get pvc问题二VM启动失败# 获取控制台输出 kubectl virt console my-vm # 检查QEMU日志 kubectl exec -it $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) -- \ cat /var/log/qemu.log # 常见错误 # - Failed to start domain: internal error: early end of file from monitor # 通常是镜像损坏重新导入镜像解决问题三网络连接异常# 进入virt-launcher调试网络 kubectl exec -it $(kubectl get pods -l kubevirt.io/domainmy-vm -o name) -- bash # 检查tap设备 ip link show # 检查iptables规则masquerade模式 iptables -t nat -L -n -v5.2 性能优化实战CPU绑核配置domain: cpu: cores: 4 sockets: 1 threads: 1 dedicatedCpuPlacement: true isolateEmulatorThread: true resources: requests: cpu: 4 memory: 8Gi limits: cpu: 4 memory: 8Gi磁盘缓存策略对比测试缓存模式4K随机读(IOPS)顺序写(MB/s)适用场景none78,000520数据库类工作负载writeback65,000680通用场景writethru58,000450数据安全优先网络性能调优参数domain: devices: interfaces: - name: default model: virtio ports: - name: http port: 8080 bandwidth: ingress: 1000 egress: 1000 tune: queues: 4 # 多队列数建议等于vCPU数6. 安全加固实践6.1 最小权限原则实施ServiceAccount权限控制apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-user rules: - apiGroups: [kubevirt.io] resources: [virtualmachines, virtualmachineinstances] verbs: [get, list, create, delete] --- apiVersion: v1 kind: ServiceAccount metadata: name: vm-operator --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-user-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubevirt-user subjects: - kind: ServiceAccount name: vm-operatorPod安全策略apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: kubevirt spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - secret hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot seLinux: rule: RunAsAny supplementalGroups: rule: MustRunAs ranges: - min: 1 max: 65535 fsGroup: rule: MustRunAs ranges: - min: 1 max: 655356.2 镜像安全扫描使用Anchore扫描VM镜像# 安装anchore-cli pip install anchorecli # 扫描镜像 anchore-cli image add quay.io/kubevirt/cirros-container-disk-demo anchore-cli image wait quay.io/kubevirt/cirros-container-disk-demo anchore-cli image vuln quay.io/kubevirt/cirros-container-disk-demo all # 集成到CI/CD anchore-cli policy add ./security-policy.json anchore-cli evaluate check \ quay.io/kubevirt/cirros-container-disk-demo \ --detail关键安全策略禁止使用latest标签基础镜像仅来自受信任仓库定期更新镜像中的软件包启用SELinux或AppArmor6.3 网络隔离方案多租户网络隔离apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: tenant-a-net annotations: k8s.v1.cni.cncf.io/resourceName: macvlan.network/kubevirt spec: config: { cniVersion: 0.3.1, name: tenant-a-net, type: macvlan, master: eth0, mode: bridge, ipam: { type: whereabouts, range: 192.168.1.0/24, exclude: [192.168.1.1/32] } }网络策略示例apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vm-isolation spec: podSelector: matchLabels: kubevirt.io/domain: my-vm policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 80 egress: - to: - podSelector: matchLabels: role: database ports: - protocol: TCP port: 3306