k8s中的微服务

一、什么是微服务

二、微服务的类型

三、IPVS模式

1、ipvs模式配置方式

（1）在所有节点中安装ipvsadm

（2）修改master节点的代理配置

（3）重启pod

四、微服务类型详解

1、clusterip

示例：

2、clusterIP中的特殊模式headless

3、nodeport

4、loadbalancer

5、metalLB

6、externalname

五、Ingress-nginx

1、ingress-nginx功能

2、部署ingress

（1）下载/上传部署文件

（2）测试ingress

3、ingress的高级用法

（1）实验环境

（2）基于路径的访问

（3）基于域名的访问

（4）建立tls加密

（5）建立auth认证

（6）rewrite重定向

六、Canary金丝雀发布

1、概念

2、Canary发布

（1）基于header（http包头）灰度

（2）基于权重的灰度发布

一、什么是微服务

用控制器来完成集群的工作负载，那么应用如何暴漏出去？需要通过微服务暴漏出去后才能被访问

Service是一组提供相同服务的Pod对外开放的接口。
借助Service，应用可以实现服务发现和负载均衡。
service默认只支持4层负载均衡能力，没有7层功能。（可以通过Ingress实现）

二、微服务的类型

微服务类型	作用描述
ClusterIP	默认值，k8s系统给service自动分配的虚拟IP，只能在集群内部访问
NodePort	将Service通过指定的Node上的端口暴露给外部，访问任意一个NodeIP:nodePort都将路由到ClusterIP
LoadBalancer	在NodePort的基础上,借助cloud provider创建一个外部的负载均衡器，并将请求转发到 NodeIP:NodePort，此模式只能在云服务器上使用
ExternalName	将服务通过 DNS CNAME 记录方式转发到指定的域名（通过 spec.externlName 设定）

示例

#生成控制器文件并建立控制器
[root@k8s-master ~]# kubectl create deployment ws --image reg.zx.org/library/myapp:v1 --replicas 2 --dry-run=client -o yaml > ws.yml
[root@k8s-master ~]# kubectl apply -f ws.yml 
deployment.apps/ws created#生成微服务yaml追加到已有yaml中
[root@k8s-master ~]# kubectl expose deployment ws --port 80 --target-port 80 --dry-run=client -o yaml >> ws.yml
[root@k8s-master ~]# vim ws.yml         #不同资源间用---隔开
[root@k8s-master ~]# cat ws.yml 
apiVersion: apps/v1
kind: Deployment
metadata:creationTimestamp: nulllabels:app: wsname: ws
spec:replicas: 2selector:matchLabels:app: wsstrategy: {}template:metadata:creationTimestamp: nulllabels:app: wsspec:containers:- image: reg.zx.org/library/myapp:v1name: myappresources: {}---apiVersion: v1
kind: Service
metadata:creationTimestamp: nulllabels:app: wsname: ws
spec:ports:- port: 80protocol: TCPtargetPort: 80selector:app: ws[root@k8s-master ~]# kubectl apply -f ws.yml 
deployment.apps/ws configured
service/ws created
[root@k8s-master ~]# kubectl get service
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
deployment   ClusterIP   10.100.147.145   <none>        80/TCP    13h
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   38h
readiness    ClusterIP   10.101.55.183    <none>        80/TCP    15h
ws           ClusterIP   10.98.35.35      <none>        80/TCP    9s[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get service -o wide
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE   SELECTOR
deployment   ClusterIP   10.100.147.145   <none>        80/TCP    13h   app=myapp
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   38h   <none>
readiness    ClusterIP   10.101.55.183    <none>        80/TCP    15h   run=readiness
ws           ClusterIP   10.98.35.35      <none>        80/TCP    67s   app=ws[root@k8s-master ~]# iptables -t nat -nL
……
Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-6GEDBFKJV7TTAFN7  tcp  --  0.0.0.0/0            10.98.35.35          /* default/ws cluster IP */ tcp dpt:80
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
……

[root@k8s-master ~]# vim ws.yml         # 添加：“type: NodePort”
[root@k8s-master ~]# cat ws.yml 
apiVersion: apps/v1
kind: Deployment
metadata:creationTimestamp: nulllabels:app: wsname: ws
spec:replicas: 2selector:matchLabels:app: wsstrategy: {}template:metadata:creationTimestamp: nulllabels:app: wsspec:containers:- image: reg.zx.org/library/myapp:v1name: myappresources: {}---apiVersion: v1
kind: Service
metadata:creationTimestamp: nulllabels:app: wsname: ws
spec:ports:- port: 80protocol: TCPtargetPort: 80selector:app: wstype: NodePort[root@k8s-master ~]# kubectl apply -f ws.yml 
deployment.apps/ws configured
service/ws configured[root@k8s-master ~]# kubectl get service
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
deployment   ClusterIP   10.100.147.145   <none>        80/TCP         13h
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        38h
readiness    ClusterIP   10.101.55.183    <none>        80/TCP         15h
ws           NodePort    10.98.35.35      <none>        80:31381/TCP   4m51s

三、IPVS模式

Service 是由 kube-proxy 组件，加上 iptables 来共同实现的

kube-proxy 通过 iptables 处理 Service 的过程，需要在宿主机上设置相当多的 iptables 规则，如果宿主机有大量的Pod，不断刷新iptables规则，会消耗大量的CPU资源

IPVS模式的service，可以使K8s集群支持更多量级的Pod

1、ipvs模式配置方式

切换ipvs模式后，kube-proxy会在宿主机上添加一个虚拟网卡：kube-ipvs0，并分配所有service IP

[root@k8s-master ~]# ip a | tail -n 20
8: veth411bce96@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 92:64:68:74:60:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 2inet6 fe80::9064:68ff:fe74:60d6/64 scope link valid_lft forever preferred_lft forever
9: vethd4d7ef26@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 66:5a:fa:34:d8:02 brd ff:ff:ff:ff:ff:ff link-netnsid 3inet6 fe80::645a:faff:fe34:d802/64 scope link valid_lft forever preferred_lft forever
11: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether 7a:63:54:5e:87:51 brd ff:ff:ff:ff:ff:ffinet 10.96.186.254/32 scope global kube-ipvs0valid_lft forever preferred_lft foreverinet 10.96.0.10/32 scope global kube-ipvs0valid_lft forever preferred_lft foreverinet 10.100.147.145/32 scope global kube-ipvs0valid_lft forever preferred_lft foreverinet 10.96.0.1/32 scope global kube-ipvs0valid_lft forever preferred_lft foreverinet 10.101.55.183/32 scope global kube-ipvs0valid_lft forever preferred_lft forever

（1）在所有节点中安装ipvsadm

yum install ipvsadm -y

（2）修改master节点的代理配置

[root@k8s-master ~]# kubectl -n kube-system edit cm kube-proxymetricsBindAddress: ""mode: "ipvs"							#设置kube-proxy使用ipvs模式nftables:

（3）重启pod

在pod运行时配置文件中采用默认配置，当改变配置文件后已经运行的pod状态不会变化，所以要重启pod

[root@k8s-master ~]# kubectl -n kube-system get  pods   | awk '/kube-proxy/{system("kubectl -n kube-system delete pods "$1)}'
pod "kube-proxy-9ftsb" deleted
pod "kube-proxy-chthh" deleted
pod "kube-proxy-dzcxh" deleted[root@k8s-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.254.200:32752 rr-> 10.244.1.65:80               Masq    1      0          0         -> 10.244.2.70:80               Masq    1      0          0         
TCP  10.96.0.1:443 rr-> 172.25.254.200:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
TCP  10.96.0.10:9153 rr
TCP  10.96.186.254:80 rr-> 10.244.1.65:80               Masq    1      0          0         -> 10.244.2.70:80               Masq    1      0          0         
TCP  10.100.147.145:80 rr
TCP  10.101.55.183:80 rr
TCP  10.244.0.1:32752 rr-> 10.244.1.65:80               Masq    1      0          0         -> 10.244.2.70:80               Masq    1      0          0   # 回收
[root@k8s-master ~]# kubectl delete -f ws.yml
deployment.apps "ws" deleted
service "ws" deleted

四、微服务类型详解

1、clusterip

特点：clusterip模式只能在集群内访问，并对集群内的pod提供健康检测和自动发现功能

示例

[root@k8s-master ~]# kubectl run testpod --image reg.zx.org/library/myapp:v1 
pod/testpod created
[root@k8s-master ~]# kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
testpod   1/1     Running   0          9s
[root@k8s-master ~]# kubectl get pods -o wide --show-labels 
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES   LABELS
testpod   1/1     Running   0          64s   10.244.1.66   k8s-node1.zx.org   <none>           <none>            run=testpod
[root@k8s-master ~]# kubectl expose pod testpod --port 80 --target-port 80 --dry-run=client -o yaml > testpod-svc.yml
[root@k8s-master ~]# vim testpod-svc.yml 
[root@k8s-master ~]# cat testpod-svc.yml 
apiVersion: v1
kind: Service
metadata:labels:run: testpodname: testpod
spec:ports:- port: 80protocol: TCPtargetPort: 80selector:run: testpodtype: ClusterIP[root@k8s-master ~]# kubectl apply -f testpod-svc.yml 
service/testpod created[root@k8s-master ~]# kubectl describe svc testpod 
Name:              testpod
Namespace:         default
Labels:            run=testpod
Annotations:       <none>
Selector:          run=testpod
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.105.248.212
IPs:               10.105.248.212
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.66:80
Session Affinity:  None
Events:            <none>
[root@k8s-master ~]# 
[root@k8s-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr-> 172.25.254.200:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
TCP  10.96.0.10:9153 rr
TCP  10.100.147.145:80 rr
TCP  10.101.55.183:80 rr
TCP  10.105.248.212:80 rr-> 10.244.1.66:80               Masq    1      0          0         
UDP  10.96.0.10:53 rr

[root@k8s-master ~]# kubectl run testpod1 --image reg.zx.org/library/myapp:v1 [root@k8s-master ~]# kubectl get pods -o wide --show-labels     # 两者标签不一致
NAME       READY   STATUS    RESTARTS   AGE    IP            NODE               NOMINATED NODE   READINESS GATES   LABELS
testpod    1/1     Running   0          7m8s   10.244.1.66   k8s-node1.zx.org   <none>           <none>            run=testpod
testpod1   1/1     Running   0          10s    10.244.2.72   k8s-node2.zx.org   <none>           <none>            run=testpod1
[root@k8s-master ~]# ipvsadm -Ln        # testpod1未在testpod调度里
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr-> 172.25.254.200:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
TCP  10.96.0.10:9153 rr
TCP  10.100.147.145:80 rr
TCP  10.101.55.183:80 rr
TCP  10.105.248.212:80 rr-> 10.244.1.66:80               Masq    1      0          0         
UDP  10.96.0.10:53 rr
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl label pod testpod1 run=testpod --overwrite     # 修改标签为tetpod
pod/testpod1 labeled
[root@k8s-master ~]# kubectl get pods -o wide --show-labels     # 标签一致
NAME       READY   STATUS    RESTARTS   AGE     IP            NODE               NOMINATED NODE   READINESS GATES   LABELS
testpod    1/1     Running   0          7m43s   10.244.1.66   k8s-node1.zx.org   <none>           <none>            run=testpod
testpod1   1/1     Running   0          45s     10.244.2.72   k8s-node2.zx.org   <none>           <none>            run=testpod
[root@k8s-master ~]# ipvsadm -Ln        
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr-> 172.25.254.200:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
TCP  10.96.0.10:9153 rr
TCP  10.100.147.145:80 rr
TCP  10.101.55.183:80 rr
TCP  10.105.248.212:80 rr            # 添加-> 10.244.1.66:80               Masq    1      0          0         -> 10.244.2.72:80               Masq    1      0          0         
UDP  10.96.0.10:53 rr

[root@k8s-master ~]# kubectl -n default get svc 
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   30m
testpod      ClusterIP   10.102.121.5   <none>        80/TCP    25s[root@k8s-master ~]#  dig testpod.default.svc.cluster.local. @10.96.0.10; <<>> DiG 9.16.23-RH <<>> testpod.default.svc.cluster.local. @10.96.0.10
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37665
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 56102b725cf1025e (echoed)
;; QUESTION SECTION:
;testpod.default.svc.cluster.local. IN	A;; ANSWER SECTION:
testpod.default.svc.cluster.local. 30 IN A	10.102.121.5;; Query time: 136 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Mon Sep 16 11:15:09 CST 2024
;; MSG SIZE  rcvd: 123# 回收
[root@k8s-master ~]# kubectl delete -f testpod-svc.yml 
service "testpod" deleted

2、clusterIP中的特殊模式headless

headless（无头服务）

对于无头 Services 并不会分配 Cluster IP，kube-proxy不会处理它们，而且平台也不会为它们进行负载均衡和路由，集群访问通过dns解析直接指向到业务pod上的IP，所有的调度有dns单独完成

[root@k8s-master ~]# vim testpod-svc.yml 
[root@k8s-master ~]# cat testpod-svc.yml 
apiVersion: v1
kind: Service
metadata:labels:run: testpodname: testpod
spec:ports:- port: 80protocol: TCPtargetPort: 80selector:run: testpodtype: ClusterIPclusterIP: None[root@k8s-master ~]# kubectl apply -f testpod-svc.yml 
service/testpod created
[root@k8s-master ~]# kubectl get service testpod 
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
testpod   ClusterIP   None         <none>        80/TCP    10s[root@k8s-master ~]# kubectl describe svc testpod 
Name:              testpod
Namespace:         default
Labels:            run=testpod
Annotations:       <none>
Selector:          run=testpod
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.66:80,10.244.2.72:80    ##直接解析到pod上
Session Affinity:  None
Events:            <none>#开启一个busyboxplus的pod测试
[root@k8s-master ~]# kubectl run  test --image reg.zx.org/library/busyboxplus:latest -it
If you don't see a command prompt, try pressing enter.
/ # nslookup testpod

3、nodeport

通过ipvs暴漏端口从而使外部主机通过master节点的对外ip:<port>来访问pod业务

其访问过程为：

[root@k8s-master ~]# vim testpod-svc.yml 
[root@k8s-master ~]# kubectl apply -f testpod-svc.yml 
service/testpod created
[root@k8s-master ~]# kubectl get service testpod 
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
testpod   NodePort   10.102.166.250   <none>        80:32171/TCP   12s
[root@k8s-master ~]# for i in {1..5}
> do
> curl 172.25.254.200:32171/hostname.html
> done

nodeport默认端口是30000-32767，超出会报错

[root@k8s-master ~]# vim testpod-svc.yml 
[root@k8s-master ~]# kubectl apply -f testpod-svc.yml 
The Service "testpod" is invalid: spec.ports[0].nodePort: Invalid value: 33333: provided port is not in the valid range. The range of valid ports is 30000-32767

如果需要使用这个范围以外的端口就需要特殊设定

[root@k8s-master ~]# vim /etc/kubernetes/manifests/kube-apiserver.yaml

添加“--service-node-port-range=“ 参数，端口范围可以自定义

修改后api-server会自动重启，等apiserver正常启动后才能操作集群

集群重启自动完成在修改完参数后全程不需要人为干预

[root@k8s-master ~]# kubectl get nodes
The connection to the server 172.25.254.200:6443 was refused - did you specify the right host or port?
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get nodes
The connection to the server 172.25.254.200:6443 was refused - did you specify the right host or port?
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get nodes
NAME                STATUS   ROLES           AGE   VERSION
k8s-master.zx.org   Ready    control-plane   40h   v1.30.0
k8s-node1.zx.org    Ready    <none>          40h   v1.30.0
k8s-node2.zx.org    Ready    <none>          40h   v1.30.0
[root@k8s-master ~]# kubectl apply -f testpod-svc.yml 
service/testpod configured
[root@k8s-master ~]# kubectl get service testpod 
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
testpod   NodePort   10.102.166.250   <none>        80:33333/TCP   10m