【k8s】ingress-nginx通过header路由到不同后端

K8S中ingress-nginx通过header路由到不同后端

背景

• 公司使用ingress-nginx作为网关的项目，需要在相同域名、uri，根据header将请求转发到不同的后端中
• 在稳定发布的情况下，ingress-nginx是没有语法直接支持根据header做转发的。但是这个可以利用灰度发布的特性实现header路由功能

准备

• 准备两个后端，后端代码如下，路由均为 /app
  • main.go

package main

import "github.com/gin-gonic/gin"

func main() {
	r := gin.Default()
	r.GET("/app", func(context *gin.Context) {
		context.JSON(200, gin.H{"message": "app1"})
	})
	r.Run(":8080")
}
1
2
3
4
5
6
7
8
9
10
11

• 使用Dockerfile构建镜像
  • 这里构建 goapp1:v1，goapp2:v1两个镜像（goapp2请将main.go修改 “message”: “app2”）

FROM golang:1.17.13
RUN mkdir -p /go/app/; \
          cd /go/app/; \
          go mod init app1;\
          GOPROXY="https://goproxy.cn,direct" go get github.com/gin-gonic/gin@v1.6.3
WORKDIR /go/app/
COPY main.go /go/app
EXPOSE 8080
CMD go run main.go
1
2
3
4
5
6
7
8
9

使用灰度发布的特性进行header的路由

• 此解决方案参考：https://v2-1.docs.kubesphere.io/docs/zh-CN/quick-start/ingress-canary/
• 注：本人使用低版本ingress-nginx，高版本的请大家自行修改不同之处
• 首先部署goapp1:v1 和 goapp2:v1 的deployment和service
  • 此为goapp1。goapp2请自行修改

apiVersion: apps/v1
kind: Deployment
metadata:
  name: goapp1
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: goapp1
  template:
    metadata:
      labels:
        app: goapp1
    spec:
      containers:
      - image: goapp1:v1
        imagePullPolicy: IfNotPresent
        name: goapp1
        ports:
        - containerPort: 80
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: goapp1
  namespace: default
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: goapp1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

• 部署稳定发布版本的ingress，路由至goapp1

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: goapp1 
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /app 
        pathType: Prefix
        backend:
          service:
            name: goapp1
            port: 
              number: 8080

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

• 部署canary版本的ingress，路由至goapp2
  • 这里可见 域名都是 test.com，uri都是 /app
  • 注解：
    • nginx.ingress.kubernetes.io/canary: “true” # 启用canary灰度发布特性
    • nginx.ingress.kubernetes.io/canary-by-header: canary # 通过header可选择是否转发至canary版本的后端

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: goapp2
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: canary
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /app 
        pathType: Prefix
        backend:
          service:
            name: goapp2
            port: 
              number: 8080
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

• 进行测试

for i in {1..20};
  # ingress-nginx的NodePort请自行查看，替换下面的端口
  do curl test.com:31132/app -H "canary: never"; # 路由至稳定版本的goapp1
  echo -e "";
done
1
2
3
4
5

for i in {1..20};
  do curl test.com:31132/app -H "canary: always"; # 路由至canary版本的goapp2
  echo -e "";
done
1
2
3
4

• 效果如下，可以看到可以通过header控制发送请求到不同后端，能够满足需求
  

通过nginx进行转发

• 第二种方法可通过在k8s集群部署一个nginx， 通过nginx进行分流
  • 流量路径如下： ingress-nginx --> nginx --> goapp1或goapp2
• 这里nginx写法有比较多，我选择最简单的通过if判断$http_my_header
• 在使用$http_my_header之前，需要对ingress-nginx和nginx添加参数，允许header中存在下划线
  • ingress-nginx

kubectl edit cm ingress-nginx-controller
------------------
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  # 添加下面这两个参数
  enable-underscores-in-headers: "true"
  ignore-invalid-headers: "false"
kind: ConfigMap
1
2
3
4
5
6
7
8
9

• 部署nginx，nginx中开启允许header下划线的参数：underscores_in_headers on;

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.24.0
        ports:
        - containerPort: 80
        volumeMounts:
          - name: nginx
            mountPath: /etc/nginx/nginx.conf
            subPath: nginx.conf
      volumes:
        - name: nginx
          configMap:
            name: nginx
            items:
            - key: nginx.conf
              path: nginx.conf
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: v1
data:
  nginx.conf: |
    user  nginx;
    worker_processes  auto;
    
    error_log  /var/log/nginx/error.log notice;
    pid        /var/run/nginx.pid;
    
    
    events {
        worker_connections  1024;
    }
    
    
    http {
    
        upstream upstream_server1 {
            server goapp1:8080;
        }
    
        upstream upstream_server2 {
            server goapp2:8080;
        }
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
    
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" "$http_my_header"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile        on;
    
        keepalive_timeout  65;
    
        server {
            underscores_in_headers on; 
            listen 80;
            server_name test.com;
    
            location /app {
              if ($http_my_header = "value1") {
                  proxy_pass http://upstream_server1;
              }
              if ($http_my_header = "value2") {
                  proxy_pass http://upstream_server2;
              }
            }
        }
    
    }

kind: ConfigMap
metadata:
  name: nginx
  namespace: default
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

• 上面的配置判断 $http_my_header是 value1 还是 value2，再转发到不同的upstream
• 测试

curl test.com/app -H "my_header:value1"
curl test.com/app -H "my_header:value2"
1
2