Kubernetes(k8s)使用问题记录和解决方法

1. Pod可以通过IP连接外网，但不能通过域名连接外网

问题记录

在Pod中ping百度，是ping不通的

root@pod:/root# ping www.baidu.com
ping: www.baidu.com: Temporary failure in name resolution
root@pod:/root#
1
2
3

我们通过解析百度的域名，获取到百度的IP。是可以ping通的

root@pod:/root# ping 104.193.88.77
PING 104.193.88.77 (104.193.88.77) 56(84) bytes of data.
64 bytes from 104.193.88.77: icmp_seq=1 ttl=127 time=178 ms
64 bytes from 104.193.88.77: icmp_seq=2 ttl=127 time=172 ms
64 bytes from 104.193.88.77: icmp_seq=3 ttl=127 time=172 ms

1
2
3
4
5
6

解决方法

添加域名解析服务器的地址，到/etc/resolv.conf中。在ping百度的域名就可以ping通了

root@pod:/root# echo "nameserver 114.114.114.114" >> /etc/resolv.conf 
root@pod:/root# 
root@pod:/root# ping www.baidu.com
PING www.a.shifen.com (14.215.177.38) 56(84) bytes of data.
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=1 ttl=127 time=26.6 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=2 ttl=127 time=25.3 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=3 ttl=127 time=26.6 ms

1
2
3
4
5
6
7
8

2. 删除coredns pod，但是deployment不能重建pod

问题记录

通过如下命令删除coredns pod，但是deployment并不能重建pod

[root@k8s-master ~]# kubectl delete pod coredns-7f74c56694-snzmv -n kube-system
pod "coredns-7f74c56694-snzmv" deleted
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl delete pod coredns-7f74c56694-whh84 -n kube-system
pod "coredns-7f74c56694-whh84" deleted
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get deploy coredns -n kube-system
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
coredns   0/2     0            0           13d
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get rs coredns-7f74c56694 -n kube-system
NAME                 DESIRED   CURRENT   READY   AGE
coredns-7f74c56694   2         0         0       13d
[root@k8s-master ~]#
1
2
3
4
5
6
7
8
9
10
11
12
13
14

查看ReplicaSet的详细信息。发现是coredns服务账号没有了

[root@k8s-master ~]# kubectl describe rs coredns-7f74c56694 -n kube-system
......省略部分......
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                     From                   Message
  ----     ------        ----                    ----                   -------
  Warning  FailedCreate  39m (x22 over 41m)      replicaset-controller  Error creating: pods "coredns-7f74c56694-" is forbidden: error looking up service account kube-system/coredns: serviceaccount "coredns" not found
[root@k8s-master ~]# 
1
2
3
4
5
6
7
8
9
10
11

解决办法

创建服务账号coredns

[root@k8s-master ~]# kubectl create serviceaccount coredns -n kube-system
serviceaccount/coredns created
[root@k8s-master ~]#
1
2
3

新建coredns-clusterrole.yaml文件，内容如下。然后创建集群角色

[root@k8s-master ~]# cat coredns-clusterrole.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:coredns
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
  - apiGroups: [""]                   
    resources: ["endpoints"]
    verbs: ["list", "watch"]
  - apiGroups: [""]                   
    resources: ["namespaces"]
    verbs: ["list", "watch"]
  - apiGroups: [""]                   
    resources: ["pods"]
    verbs: ["list", "watch"]
  - apiGroups: [""]                   
    resources: ["services"]
    verbs: ["list", "watch"]
  - apiGroups: ["discovery.k8s.io"]                   
    resources: ["endpointslices"]
    verbs: ["list", "watch"]

[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f coredns-clusterrole.yaml 
clusterrole.rbac.authorization.k8s.io/system:coredns created
[root@k8s-master ~]# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

然后绑定角色，给coredns服务账号授权

[root@k8s-master ~]# kubectl create clusterrolebinding system:coredns2 --clusterrole=system:coredns  --serviceaccount=kube-system:coredns
clusterrolebinding.rbac.authorization.k8s.io/system:coredns2 created
[root@k8s-master ~]#
1
2
3

3. 重建的coredns Pod，没有configmap coredns

问题记录
查看coredns pod详细信息，如下所示

[root@k8s-master ~]# kubectl describe pod coredns-7f74c56694-lmkbf -n kube-system
......省略部分......
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    25m                  default-scheduler  Successfully assigned kube-system/coredns-7f74c56694-lmkbf to k8s-node1
  Warning  FailedMount  19m (x11 over 25m)   kubelet            MountVolume.SetUp failed for volume "config-volume" : configmap "coredns" not found
  Warning  FailedMount  18m (x3 over 23m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[config-volume], unattached volumes=[config-volume kube-api-access-ctnpg]: timed out waiting for the condition
[root@k8s-master ~]#
1
2
3
4
5
6
7
8
9
10
11
12
13
14

解决办法

[root@k8s-master ~]# cat coredns-configmap.yaml 
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2022-05-12T04:54:57Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "239"
  uid: bcb58086-8b67-448a-88d1-6cf99c1fb621
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f coredns-configmap.yaml 
configmap/coredns created
[root@k8s-master ~]# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

4. Pod内不能进行域名解析，kube-dns Service不存在

问题记录
在Pod内进行ping百度的IP可以ping通，ping百度的域名不能ping通

root@pod:/# ping 103.235.46.39
PING 103.235.46.39 (103.235.46.39): 56 data bytes
64 bytes from 103.235.46.39: icmp_seq=0 ttl=127 time=235.682 ms
64 bytes from 103.235.46.39: icmp_seq=1 ttl=127 time=244.924 ms
64 bytes from 103.235.46.39: icmp_seq=2 ttl=127 time=262.843 ms
root@pod:/# 
root@pod:/# ping www.baidu.com
ping: unknown host
root@pod:/#
1
2
3
4
5
6
7
8
9

查看kube-dns Service，发现不存在

[root@k8s-master ~]# kubectl get svc kube-dns -n kube-system
Error from server (NotFound): services "kube-dns" not found
[root@k8s-master ~]# 
1
2
3

解决办法
新建kube-dns.yaml文件，内容如下。然后创建kube-dns Service

[root@k8s-master ~]# cat kube-dns.yaml 
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  creationTimestamp: "2022-05-12T04:54:57Z"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: kube-dns
  namespace: kube-system
  resourceVersion: "245"
  uid: 4bafe2a2-14d8-4db2-81ab-6d826d93a454
spec:
  clusterIP: 10.96.0.10
  clusterIPs:
  - 10.96.0.10
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  - name: metrics
    port: 9153
    protocol: TCP
    targetPort: 9153
  selector:
    k8s-app: kube-dns
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f kube-dns.yaml 
service/kube-dns created
[root@k8s-master ~]#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

再次ping百度的域名，就可以ping通了