• 云原生（二十八） | Kubernetes篇之自建高可用k8s集群搭建

    文末有惊喜

    文章目录

    自建高可用k8s集群搭建

    一、所有节点基础环境

    1、环境准备与内核升级

    2、安装Docker

    二、PKI

    三、证书工具准备

    1、下载证书工具

    2、ca根配置

    3、ca签名请求

    4、生成证书

    5、k8s集群是如何使用证书的

    四、etcd高可用搭建

    1、etcd文档

    2、下载etcd

    3、etcd证书

    4、etcd高可用安装

    五、k8s组件与证书

    1、K8s离线安装包

    2、master节点准备

    3、apiserver 证书生成

    4、front-proxy证书生成

    5、controller-manage证书生成与配置

    6、scheduler证书生成与配置

    7、admin证书生成与配置

    8、ServiceAccount Key生成

    9、发送证书到其他节点

    六、高可用配置

     七、组件启动

    1、所有master执行

    2、配置apiserver服务

    3、配置controller-manager服务

    4、配置scheduler

    八、TLS与引导启动原理

    1、master1配置bootstrap

    2、master1设置kubectl执行权限

    3、创建集群引导权限文件

    九、引导Node节点启动

    1、发送核心证书到节点

    2、所有节点配置kubelet

    3、kube-proxy配置

    十、部署calico

    十一、部署coreDNS

    十二、给机器打上role标签

    十三、集群验证

    自建高可用k8s集群搭建

    一、所有节点基础环境

    192.168.0.x ： 为机器的网段

    10.96.0.0/16: 为Service网段

    196.16.0.0/16: 为Pod网段

    1、环境准备与内核升级

    先升级所有机器内核

    1. #我的机器版本
    2. cat /etc/redhat-release 
    3. # CentOS Linux release 7.9.2009 (Core)
    4. #修改域名，一定不是localhost
    5. hostnamectl set-hostname k8s-xxx
    6.  
    7.  
    8. #集群规划
    9. k8s-master1  k8s-master2  k8s-master3 k8s-master-lb k8s-node01  k8s-node02 ... k8s-nodeN
    10.  
    11. # 每个机器准备域名
    12. vi /etc/hosts
    13. 192.168.0.10 k8s-master1
    14. 192.168.0.11 k8s-master2
    15. 192.168.0.12 k8s-master3
    16. 192.168.0.13 k8s-node1
    17. 192.168.0.14 k8s-node2
    18. 192.168.0.15 k8s-node3
    19. 192.168.0.250 k8s-master-lb # 非高可用，可以不用这个。这个使用keepalive配置
    1. # 关闭selinux
    2. setenforce 0
    3. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
    4. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
    1. # 关闭swap
    2. swapoff -a && sysctl -w vm.swappiness=0
    3. sed -ri 's/.*swap.*/#&/' /etc/fstab
    1. #修改limit
    2. ulimit -SHn 65535
    3. vi /etc/security/limits.conf
    4. # 末尾添加如下内容
    5. * soft nofile 655360
    6. * hard nofile 131072
    7. * soft nproc 655350
    8. * hard nproc 655350
    9. * soft memlock unlimited
    10. * hard memlock unlimited
    1. #为了方便以后操作配置ssh免密连接，master1运行
    2. ssh-keygen -t rsa
    3.  
    4. for i in k8s-master1 k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
    1. #安装后续用的一些工具
    2. yum install wget git jq psmisc net-tools yum-utils device-mapper-persistent-data lvm2  -y
    1. # 所有节点
    2. # 安装ipvs工具,方便以后操作ipvs，ipset，conntrack等
    3. yum install ipvsadm ipset sysstat conntrack libseccomp -y
    4. # 所有节点配置ipvs模块,执行以下命令，在内核4.19+版本改为nf_conntrack， 4.18下改为nf_conntrack_ipv4
    5. modprobe -- ip_vs
    6. modprobe -- ip_vs_rr
    7. modprobe -- ip_vs_wrr
    8. modprobe -- ip_vs_sh
    9. modprobe -- nf_conntrack
    10.  
    11. #修改ipvs配置，加入以下内容
    12. vi /etc/modules-load.d/ipvs.conf
    13.  
    14. ip_vs
    15. ip_vs_lc
    16. ip_vs_wlc
    17. ip_vs_rr
    18. ip_vs_wrr
    19. ip_vs_lblc
    20. ip_vs_lblcr
    21. ip_vs_dh
    22. ip_vs_sh
    23. ip_vs_fo
    24. ip_vs_nq
    25. ip_vs_sed
    26. ip_vs_ftp
    27. ip_vs_sh
    28. nf_conntrack
    29. ip_tables
    30. ip_set
    31. xt_set
    32. ipt_set
    33. ipt_rpfilter
    34. ipt_REJECT
    35. ipip
    36.  
    37. # 执行命令
    38. systemctl enable --now systemd-modules-load.service  #--now = enable+start
    39.  
    40. #检测是否加载
    41. lsmod | grep -e ip_vs -e nf_conntrack
    1. ## 所有节点
    2. cat <<EOF > /etc/sysctl.d/k8s.conf
    3. net.ipv4.ip_forward = 1
    4. net.bridge.bridge-nf-call-iptables = 1
    5. net.bridge.bridge-nf-call-ip6tables = 1
    6. fs.may_detach_mounts = 1
    7. vm.overcommit_memory=1
    8. net.ipv4.conf.all.route_localnet = 1
    10. vm.panic_on_oom=0
    11. fs.inotify.max_user_watches=89100
    12. fs.file-max=52706963
    13. fs.nr_open=52706963
    14. net.netfilter.nf_conntrack_max=2310720
    16. net.ipv4.tcp_keepalive_time = 600
    17. net.ipv4.tcp_keepalive_probes = 3
    18. net.ipv4.tcp_keepalive_intvl =15
    19. net.ipv4.tcp_max_tw_buckets = 36000
    20. net.ipv4.tcp_tw_reuse = 1
    21. net.ipv4.tcp_max_orphans = 327680
    22. net.ipv4.tcp_orphan_retries = 3
    23. net.ipv4.tcp_syncookies = 1
    24. net.ipv4.tcp_max_syn_backlog = 16768
    25. net.ipv4.ip_conntrack_max = 65536
    26. net.ipv4.tcp_timestamps = 0
    27. net.core.somaxconn = 16768
    28. EOF
    29. sysctl --system
    1. # 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载
    2. reboot
    3. lsmod | grep -e ip_vs -e nf_conntrack

    2、安装Docker

    1. # 安装docker
    2. yum remove docker*
    3. yum install -y yum-utils
    4. yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    5. yum install -y docker-ce-19.03.9  docker-ce-cli-19.03.9 containerd.io-1.4.4
    1. # 修改docker配置,新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd
    2. mkdir /etc/docker
    3. cat > /etc/docker/daemon.json <<EOF
    4. {
    5.   "exec-opts": ["native.cgroupdriver=systemd"],
    6.   "registry-mirrors": ["https://82m9ayutr63.mirror.aliyuncs.com"]
    7. }
    8. EOF
    9. systemctl daemon-reload && systemctl enable --now docker
    1. #也可以自己下载rpm离线包进行安装
    2. http://mirrors.aliyun.com/docker-ce/linux/centos/7.9/x86_64/stable/Packages/
    3. yum localinstall xxxx

    二、PKI

    百度百科：公钥基础设施_百度百科

    Kubernetes 需要 PKI 才能执行以下操作：

    • Kubelet 的客户端证书，用于 API 服务器身份验证

    • API 服务器端点的证书

    • 集群管理员的客户端证书，用于 API 服务器身份认证

    • API 服务器的客户端证书，用于和 Kubelet 的会话

    • API 服务器的客户端证书，用于和 etcd 的会话

    • 控制器管理器的客户端证书/kubeconfig，用于和 API 服务器的会话

    • 调度器的客户端证书/kubeconfig，用于和 API 服务器的会话

    • 前端代理的客户端及服务端证书

    说明： 只有当你运行 kube-proxy 并要支持扩展 API 服务器 时，才需要 front-proxy 证书

    etcd 还实现了双向 TLS 来对客户端和对其他对等节点进行身份验证

    PKI 证书和要求 | Kubernetes

    三、证书工具准备

    1. # 准备文件夹存放所有证书信息。看看kubeadm 如何组织有序的结构的
    2. # 三个节点都执行
    3. mkdir -p /etc/kubernetes/pki

    1、下载证书工具

    1. # 下载cfssl核心组件
    2. wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64
    3. wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64
    4. wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64
    5.  
    6. #授予执行权限
    7. chmod +x cfssl*
    8.  
    9. #批量重命名
    10. for name in `ls cfssl*`; do mv $name ${name%_1.5.0_linux_amd64};  done
    11.  
    12. #移动到文件
    13. mv cfssl* /usr/bin

    2、ca根配置

    ca-config.json

    1. mkdir -p /etc/kubernetes/pki
    2. cd /etc/kubernetes/pki
    3. vi ca-config.json
    1. {
    2.     "signing": {
    3.         "default": {
    4.             "expiry": "87600h"
    5.         },
    6.         "profiles": {
    7.             "server": {
    8.                 "expiry": "87600h",
    9.                 "usages": [
    10.                     "signing",
    11.                     "key encipherment",
    12.                     "server auth"
    13.                 ]
    14.             },
    15.             "client": {
    16.                 "expiry": "87600h",
    17.                 "usages": [
    18.                     "signing",
    19.                     "key encipherment",
    20.                     "client auth"
    21.                 ]
    22.             },
    23.             "peer": {
    24.                 "expiry": "87600h",
    25.                 "usages": [
    26.                     "signing",
    27.                     "key encipherment",
    28.                     "server auth",
    29.                     "client auth"
    30.                 ]
    31.             },
    32.             "kubernetes": {
    33.                 "expiry": "87600h",
    34.                 "usages": [
    35.                     "signing",
    36.                     "key encipherment",
    37.                     "server auth",
    38.                     "client auth"
    39.                 ]
    40.             },
    41.             "etcd": {
    42.                 "expiry": "87600h",
    43.                 "usages": [
    44.                     "signing",
    45.                     "key encipherment",
    46.                     "server auth",
    47.                     "client auth"
    48.                 ]
    49.             }
    50.         }
    51.     }
    52. }

    3、ca签名请求

    CSR是Certificate Signing Request的英文缩写，即证书签名请求文件

    ca-csr.json

    vi /etc/kubernetes/pki/ca-csr.json
    1. {
    2.   "CN": "kubernetes",
    3.   "key": {
    4.     "algo": "rsa",
    5.     "size": 2048
    6.   },
    7.   "names": [
    8.     {
    9.       "C": "CN",
    10.       "ST": "Beijing",
    11.       "L": "Beijing",
    12.       "O": "Kubernetes",
    13.       "OU": "Kubernetes"
    14.     }
    15.   ],
    16.   "ca": {
    17.     "expiry": "87600h"
    18.   }
    19. }

    • CN(Common Name):

      • 公用名（Common Name）必须填写，一般可以是网站域

    • O(Organization):

      • Organization（组织名）是必须填写的，如果申请的是OV、EV型证书，组织名称必须严格和企业在政府登记名称一致，一般需要和营业执照上的名称完全一致。不可以使用缩写或者商标。如果需要使用英文名称，需要有DUNS编码或者律师信证明。

    • OU(Organization Unit)

      • OU单位部门，这里一般没有太多限制，可以直接填写IT DEPT等皆可。

    • C(City)

      • City是指申请单位所在的城市。

    • ST(State/Province)

      • ST是指申请单位所在的省份。

    • C(Country Name）

      • C是指国家名称，这里用的是两位大写的国家代码，中国是CN。

    4、生成证书

    生成ca证书和私钥

    1. cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
    2. # ca.csr ca.pem(ca公钥) ca-key.pem(ca私钥,妥善保管)


    5、k8s集群是如何使用证书的

    参考官方文档：PKI 证书和要求 | Kubernetes

    四、etcd高可用搭建

    1、etcd文档

    etcd示例：Demo | etcd 参照示例学习etcd使用

    etcd构建：Install | etcd 参照etcd-k8s集群量规划指南，大家参照这个标准建立集群

    etcd部署：Operations guide | etcd 参照部署手册，学习etcd配置和集群部署

    2、下载etcd

    1. # 给所有master节点，发送etcd包准备部署etcd高可用
    2. wget https://github.com/etcd-io/etcd/releases/download/v3.4.16/etcd-v3.4.16-linux-amd64.tar.gz
    3.  
    4. ## 复制到其他节点
    5. for i in k8s-master1 k8s-master2 k8s-master3;do scp etcd-* root@$i:/root/;done
    6.  
    7.  
    8. ## 解压到 /usr/local/bin
    9. tar -zxvf etcd-v3.4.16-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.4.16-linux-amd64/etcd{,ctl}
    10.  
    11.  
    12. ##验证
    13. etcdctl #只要有打印就ok

    3、etcd证书

    Hardware recommendations | etcd安装参考 ：Hardware recommendations | etcd

    生成etcd证书

    etcd-ca-csr.json

    1. {
    2.   "CN": "etcd",
    3.   "key": {
    4.     "algo": "rsa",
    5.     "size": 2048
    6.   },
    7.   "names": [
    8.     {
    9.       "C": "CN",
    10.       "ST": "Beijing",
    11.       "L": "Beijing",
    12.       "O": "etcd",
    13.       "OU": "etcd"
    14.     }
    15.   ],
    16.   "ca": {
    17.     "expiry": "87600h"
    18.   }
    19. }
    1. # 生成etcd根ca证书
    2. cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/ca -

    etcd-itdachang-csr.json

    1. {
    2.     "CN": "etcd-itdachang",
    3.     "key": {
    4.         "algo": "rsa",
    5.         "size": 2048
    6.     },
    7.     "hosts": [  
    8.         "127.0.0.1",
    9.         "k8s-master1",
    10.         "k8s-master2",
    11.         "k8s-master3",
    12.         "192.168.0.10",
    13.         "192.168.0.11",
    14.         "192.168.0.12"
    15.     ],
    16.     "names": [
    17.         {
    18.             "C": "CN",
    19.             "L": "beijing",
    20.             "O": "etcd",
    21.             "ST": "beijing",
    22.             "OU": "System"
    23.         }
    24.     ]
    25. }
    26.  
    27. // 注意：hosts用自己的主机名和ip
    28. // 也可以在签发的时候再加上 -hostname=127.0.0.1,k8s-master1,k8s-master2,k8s-master3,
    29. // 可以指定受信的主机列表
    30. //    "hosts": [
    31. //        "k8s-master1",
    32. //        "www.example.net"
    33. //    ],
    1. # 签发itdachang的etcd证书
    2. cfssl gencert \
    3.    -ca=/etc/kubernetes/pki/etcd/ca.pem \
    4.    -ca-key=/etc/kubernetes/pki/etcd/ca-key.pem \
    5.    -config=/etc/kubernetes/pki/ca-config.json \
    6.    -profile=etcd \
    7.    etcd-itdachang-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/etcd
    1. # 把生成的etcd证书，复制给其他机器
    2.  
    3. for i in k8s-master2 k8s-master3;do scp -r /etc/kubernetes/pki/etcd root@$i:/etc/kubernetes/pki;done

    4、etcd高可用安装

    etcd配置文件示例： Configuration flags | etcd

    etcd高可用安装示例： Clustering Guide | etcd

    为了保证启动配置一致性，我们编写etcd配置文件，并将etcd做成service启动

    1. # etcd yaml示例。
    2. # This is the configuration file for the etcd server.
    3.  
    4. # Human-readable name for this member.
    5. name: 'default'
    6. # Path to the data directory.
    7. data-dir:
    8. # Path to the dedicated wal directory.
    9. wal-dir:
    10. # Number of committed transactions to trigger a snapshot to disk.
    11. snapshot-count: 10000
    12. # Time (in milliseconds) of a heartbeat interval.
    13. heartbeat-interval: 100
    14. # Time (in milliseconds) for an election to timeout.
    15. election-timeout: 1000
    16. # Raise alarms when backend size exceeds the given quota. 0 means use the
    17. # default quota.
    18. quota-backend-bytes: 0
    19. # List of comma separated URLs to listen on for peer traffic.
    20. listen-peer-urls: http://localhost:2380
    21. # List of comma separated URLs to listen on for client traffic.
    22. listen-client-urls: http://localhost:2379
    23. # Maximum number of snapshot files to retain (0 is unlimited).
    24. max-snapshots: 5
    25. # Maximum number of wal files to retain (0 is unlimited).
    26. max-wals: 5
    27. # Comma-separated white list of origins for CORS (cross-origin resource sharing).
    28. cors:
    29. # List of this member's peer URLs to advertise to the rest of the cluster.
    30. # The URLs needed to be a comma-separated list.
    31. initial-advertise-peer-urls: http://localhost:2380
    32. # List of this member's client URLs to advertise to the public.
    33. # The URLs needed to be a comma-separated list.
    34. advertise-client-urls: http://localhost:2379
    35. # Discovery URL used to bootstrap the cluster.
    36. discovery:
    37. # Valid values include 'exit', 'proxy'
    38. discovery-fallback: 'proxy'
    39. # HTTP proxy to use for traffic to discovery service.
    40. discovery-proxy:
    41. # DNS domain used to bootstrap initial cluster.
    42. discovery-srv:
    43. # Initial cluster configuration for bootstrapping.
    44. initial-cluster:
    45. # Initial cluster token for the etcd cluster during bootstrap.
    46. initial-cluster-token: 'etcd-cluster'
    47. # Initial cluster state ('new' or 'existing').
    48. initial-cluster-state: 'new'
    49. # Reject reconfiguration requests that would cause quorum loss.
    50. strict-reconfig-check: false
    51. # Accept etcd V2 client requests
    52. enable-v2: true
    53. # Enable runtime profiling data via HTTP server
    54. enable-pprof: true
    55. # Valid values include 'on', 'readonly', 'off'
    56. proxy: 'off'
    57. # Time (in milliseconds) an endpoint will be held in a failed state.
    58. proxy-failure-wait: 5000
    59. # Time (in milliseconds) of the endpoints refresh interval.
    60. proxy-refresh-interval: 30000
    61. # Time (in milliseconds) for a dial to timeout.
    62. proxy-dial-timeout: 1000
    63. # Time (in milliseconds) for a write to timeout.
    64. proxy-write-timeout: 5000
    65. # Time (in milliseconds) for a read to timeout.
    66. proxy-read-timeout: 0
    67. client-transport-security:
    68.   # Path to the client server TLS cert file.
    69.   cert-file:
    70.   # Path to the client server TLS key file.
    71.   key-file:
    72.   # Enable client cert authentication.
    73.   client-cert-auth: false
    74.   # Path to the client server TLS trusted CA cert file.
    75.   trusted-ca-file:
    76.   # Client TLS using generated certificates
    77.   auto-tls: false
    78. peer-transport-security:
    79.   # Path to the peer server TLS cert file.
    80.   cert-file:
    81.   # Path to the peer server TLS key file.
    82.   key-file:
    83.   # Enable peer client cert authentication.
    84.   client-cert-auth: false
    85.   # Path to the peer server TLS trusted CA cert file.
    86.   trusted-ca-file:
    87.   # Peer TLS using generated certificates.
    88.   auto-tls: false
    89. # Enable debug-level logging for etcd.
    90. debug: false
    91. logger: zap
    92. # Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd.
    93. log-outputs: [stderr]
    94. # Force to create a new one member cluster.
    95. force-new-cluster: false
    96. auto-compaction-mode: periodic
    97. auto-compaction-retention: "1"

    三个etcd机器都创建 /etc/etcd 目录，准备存储etcd配置信息  

    1. #三个master执行
    2. mkdir -p /etc/etcd
    vi /etc/etcd/etcd.yaml
    1. # 我们的yaml
    2. name: 'etcd-master3'  #每个机器可以写自己的域名,不能重复
    3. data-dir: /var/lib/etcd
    4. wal-dir: /var/lib/etcd/wal
    5. snapshot-count: 5000
    6. heartbeat-interval: 100
    7. election-timeout: 1000
    8. quota-backend-bytes: 0
    9. listen-peer-urls: 'https://192.168.0.12:2380'  # 本机ip+2380端口，代表和集群通信
    10. listen-client-urls: 'https://192.168.0.12:2379,http://127.0.0.1:2379' #改为自己的
    11. max-snapshots: 3
    12. max-wals: 5
    13. cors:
    14. initial-advertise-peer-urls: 'https://192.168.0.12:2380' #自己的ip
    15. advertise-client-urls: 'https://192.168.0.12:2379'  #自己的ip
    16. discovery:
    17. discovery-fallback: 'proxy'
    18. discovery-proxy:
    19. discovery-srv:
    20. initial-cluster: 'etcd-master1=https://192.168.0.10:2380,etcd-master2=https://192.168.0.11:2380,etcd-master3=https://192.168.0.12:2380' #这里不一样
    21. initial-cluster-token: 'etcd-k8s-cluster'
    22. initial-cluster-state: 'new'
    23. strict-reconfig-check: false
    24. enable-v2: true
    25. enable-pprof: true
    26. proxy: 'off'
    27. proxy-failure-wait: 5000
    28. proxy-refresh-interval: 30000
    29. proxy-dial-timeout: 1000
    30. proxy-write-timeout: 5000
    31. proxy-read-timeout: 0
    32. client-transport-security:
    33.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
    34.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
    35.   client-cert-auth: true
    36.   trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.pem'
    37.   auto-tls: true
    38. peer-transport-security:
    39.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
    40.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
    41.   peer-client-cert-auth: true
    42.   trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.pem'
    43.   auto-tls: true
    44. debug: false
    45. log-package-levels:
    46. log-outputs: [default]
    47. force-new-cluster: false

    三台机器的etcd做成service，开机启动  

    1. vi /usr/lib/systemd/system/etcd.service
    2.  
    3. [Unit]
    4. Description=Etcd Service
    5. Documentation=https://etcd.io/docs/v3.4/op-guide/clustering/
    6. After=network.target
    7.  
    8. [Service]
    9. Type=notify
    10. ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.yaml
    11. Restart=on-failure
    12. RestartSec=10
    13. LimitNOFILE=65536
    14.  
    15. [Install]
    16. WantedBy=multi-user.target
    17. Alias=etcd3.service
    1. # 加载&开机启动
    2. systemctl daemon-reload
    3. systemctl enable --now etcd
    4.  
    5. # 启动有问题,使用 journalctl -u 服务名排查
    6. journalctl -u etcd

    测试etcd访问

    1. # 查看etcd集群状态
    2. etcdctl --endpoints="192.168.0.10:2379,192.168.0.11:2379,192.168.0.12:2379" --cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
    3.  
    4. # 以后测试命令
    5. export ETCDCTL_API=3
    6. HOST_1=192.168.0.10
    7. HOST_2=192.168.0.11
    8. HOST_3=192.168.0.12
    9. ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
    10.  
    11. ## 导出环境变量，方便测试，参照https://github.com/etcd-io/etcd/tree/main/etcdctl
    12. export ETCDCTL_DIAL_TIMEOUT=3s
    13. export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.pem
    14. export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/etcd.pem
    15. export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/etcd-key.pem
    16. export ETCDCTL_ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
    17. # 自动用环境变量定义的证书位置
    18. etcdctl  member list --write-out=table
    19.  
    20. #如果没有环境变量就需要如下方式调用
    21. etcdctl --endpoints=$ENDPOINTS --cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem member list --write-out=table
    22.  
    23.  
    24. ## 更多etcdctl命令，https://etcd.io/docs/v3.4/demo/#access-etcd

    五、k8s组件与证书

    1、K8s离线安装包

    https://github.com/kubernetes/kubernetes 找到changelog对应版本

    1. # 下载k8s包
    2. wget https://dl.k8s.io/v1.21.1/kubernetes-server-linux-amd64.tar.gz

    2、master节点准备

    1. # 把kubernetes把复制给master所有节点
    2. for i in k8s-master1 k8s-master2 k8s-master3  k8s-node1 k8s-node2 k8s-node3;do scp kubernetes-server-* root@$i:/root/;done
    1. #所有master节点解压kubelet，kubectl等到 /usr/local/bin。
    2. tar -xvf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
    3.  
    4.  
    5. #master需要全部组件，node节点只需要 /usr/local/bin kubelet、kube-proxy

    3、apiserver 证书生成

    3.1、apiserver-csr.json

    1. //10.96.0. 为service网段。可以自定义 如： 66.66.0.1
    2. // 192.168.0.250： 是我准备的负载均衡器地址（负载均衡可以自己搭建，也可以购买云厂商lb。）
    3. {
    4.     "CN": "kube-apiserver",
    5.     "hosts": [
    6.       "10.96.0.1",
    7.       "127.0.0.1",
    8.       "192.168.0.250",
    9.       "192.168.0.10",
    10.       "192.168.0.11",
    11.       "192.168.0.12",
    12.       "192.168.0.13",
    13.       "192.168.0.14",
    14.       "192.168.0.15",
    15.       "192.168.0.16",
    16.       "kubernetes",
    17.       "kubernetes.default",
    18.       "kubernetes.default.svc",
    19.       "kubernetes.default.svc.cluster",
    20.       "kubernetes.default.svc.cluster.local"
    21.     ],
    22.     "key": {
    23.         "algo": "rsa",
    24.         "size": 2048
    25.     },
    26.     "names": [
    27.         {
    28.             "C": "CN",
    29.             "L": "BeiJing",
    30.             "ST": "BeiJing",
    31.             "O": "Kubernetes",
    32.             "OU": "Kubernetes"
    33.         }
    34.     ]
    35. }

    3.2、生成apiserver证书

    1. # 192.168.0.是k8s service的网段，如果说需要更改k8s service网段，那就需要更改192.168.0.1，
    2. # 如果不是高可用集群，10.103.236.236为Master01的IP
    3. #先生成CA机构
    4. vi ca-csr.json
    5. {
    6.   "CN": "kubernetes",
    7.   "key": {
    8.     "algo": "rsa",
    9.     "size": 2048
    10.   },
    11.   "names": [
    12.     {
    13.       "C": "CN",
    14.       "ST": "Beijing",
    15.       "L": "Beijing",
    16.       "O": "Kubernetes",
    17.       "OU": "Kubernetes"
    18.     }
    19.   ],
    20.   "ca": {
    21.     "expiry": "87600h"
    22.   }
    23. }
    24.  
    25.  
    26. cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
    27.  
    28. cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem   -ca-key=/etc/kubernetes/pki/ca-key.pem   -config=/etc/kubernetes/pki/ca-config.json   -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

    4、front-proxy证书生成

    官方文档：配置聚合层 | Kubernetes

    注意：front-proxy不建议用新的CA机构签发证书，可能导致通过他代理的组件如metrics-server权限不可用。

    如果用新的，api-server配置添加 --requestheader-allowed-names=front-proxy-client

     4.1、front-proxy-ca-csr.json

    front-proxy根ca

    1. vi front-proxy-ca-csr.json
    2. {
    3.   "CN": "kubernetes",
    4.   "key": {
    5.      "algo": "rsa",
    6.      "size": 2048
    7.   }
    8. }
    1. #front-proxy 根ca生成
    2. cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

    4.2、front-proxy-client证书

    vi  front-proxy-client-csr.json  #准备申请client客户端
    1. {
    2.   "CN": "front-proxy-client",
    3.   "key": {
    4.      "algo": "rsa",
    5.      "size": 2048
    6.   }
    7. }
    1. #生成front-proxy-client 证书
    2. cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
    3.  
    4. #忽略警告，毕竟我们不是给网站生成的

    5、controller-manage证书生成与配置

    5.1、controller-manager-csr.json

    vi controller-manager-csr.json
    1. {
    2.   "CN": "system:kube-controller-manager",
    3.   "key": {
    4.     "algo": "rsa",
    5.     "size": 2048
    6.   },
    7.   "names": [
    8.     {
    9.       "C": "CN",
    10.       "ST": "Beijing",
    11.       "L": "Beijing",
    12.       "O": "system:kube-controller-manager",
    13.       "OU": "Kubernetes"
    14.     }
    15.   ]
    16. }

    5.2、生成证书

    1. cfssl gencert \
    2.    -ca=/etc/kubernetes/pki/ca.pem \
    3.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
    4.    -config=ca-config.json \
    5.    -profile=kubernetes \
    6.   controller-manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

    5.3、生成配置

    1. # 注意，如果不是高可用集群，192.168.0.250:6443改为master01的地址，6443为apiserver的默认端口
    2. # set-cluster：设置一个集群项，
    3. kubectl config set-cluster kubernetes \
    4.      --certificate-authority=/etc/kubernetes/pki/ca.pem \
    5.      --embed-certs=true \
    6.      --server=https://192.168.0.250:6443 \
    7.      --kubeconfig=/etc/kubernetes/controller-manager.conf
    8.  
    9. # 设置一个环境项，一个上下文
    10. kubectl config set-context system:kube-controller-manager@kubernetes \
    11.     --cluster=kubernetes \
    12.     --user=system:kube-controller-manager \
    13.     --kubeconfig=/etc/kubernetes/controller-manager.conf
    14.  
    15. # set-credentials 设置一个用户项
    16.  
    17. kubectl config set-credentials system:kube-controller-manager \
    18.      --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
    19.      --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
    20.      --embed-certs=true \
    21.      --kubeconfig=/etc/kubernetes/controller-manager.conf
    22.  
    23.  
    24. # 使用某个环境当做默认环境
    25.  
    26. kubectl config use-context system:kube-controller-manager@kubernetes \
    27.      --kubeconfig=/etc/kubernetes/controller-manager.conf
    28.      
    29.      
    30. # 后来也用来自动批复kubelet证书

    6、scheduler证书生成与配置

    6.1、scheduler-csr.json

    vi scheduler-csr.json
    1. {
    2.   "CN": "system:kube-scheduler",
    3.   "key": {
    4.     "algo": "rsa",
    5.     "size": 2048
    6.   },
    7.   "names": [
    8.     {
    9.       "C": "CN",
    10.       "ST": "Beijing",
    11.       "L": "Beijing",
    12.       "O": "system:kube-scheduler",
    13.       "OU": "Kubernetes"
    14.     }
    15.   ]
    16. }

    6.2、签发证书

    1. cfssl gencert \
    2.    -ca=/etc/kubernetes/pki/ca.pem \
    3.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
    4.    -config=/etc/kubernetes/pki/ca-config.json \
    5.    -profile=kubernetes \
    6.    scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

    6.3、生成配置

    1. # 注意，如果不是高可用集群，192.168.0.250:6443 改为master01的地址，6443是api-server默认端口
    2.  
    3. kubectl config set-cluster kubernetes \
    4.      --certificate-authority=/etc/kubernetes/pki/ca.pem \
    5.      --embed-certs=true \
    6.      --server=https://192.168.0.250:6443 \
    7.      --kubeconfig=/etc/kubernetes/scheduler.conf
    8.  
    9.  
    10. kubectl config set-credentials system:kube-scheduler \
    11.      --client-certificate=/etc/kubernetes/pki/scheduler.pem \
    12.      --client-key=/etc/kubernetes/pki/scheduler-key.pem \
    13.      --embed-certs=true \
    14.      --kubeconfig=/etc/kubernetes/scheduler.conf
    15.  
    16. kubectl config set-context system:kube-scheduler@kubernetes \
    17.      --cluster=kubernetes \
    18.      --user=system:kube-scheduler \
    19.      --kubeconfig=/etc/kubernetes/scheduler.conf
    20.  
    21.  
    22. kubectl config use-context system:kube-scheduler@kubernetes \
    23.      --kubeconfig=/etc/kubernetes/scheduler.conf
    24.  
    25. #k8s集群安全操作相关

    7、admin证书生成与配置

    7.1、admin-csr.json

    vi admin-csr.json
    1. {
    2.   "CN": "admin",
    3.   "key": {
    4.     "algo": "rsa",
    5.     "size": 2048
    6.   },
    7.   "names": [
    8.     {
    9.       "C": "CN",
    10.       "ST": "Beijing",
    11.       "L": "Beijing",
    12.       "O": "system:masters",
    13.       "OU": "Kubernetes"
    14.     }
    15.   ]
    16. }

    7.2、生成证书

    1. cfssl gencert \
    2.    -ca=/etc/kubernetes/pki/ca.pem \
    3.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
    4.    -config=/etc/kubernetes/pki/ca-config.json \
    5.    -profile=kubernetes \
    6.    admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

    7.3、生成配置

    1. # 注意，如果不是高可用集群，192.168.0.250:6443改为master01的地址，6443为apiserver的默认端口
    2. kubectl config set-cluster kubernetes \
    3. --certificate-authority=/etc/kubernetes/pki/ca.pem \
    4. --embed-certs=true \
    5. --server=https://192.168.0.250:6443 \
    6. --kubeconfig=/etc/kubernetes/admin.conf
    7.  
    8.  
    9. kubectl config set-credentials kubernetes-admin \
    10. --client-certificate=/etc/kubernetes/pki/admin.pem \
    11. --client-key=/etc/kubernetes/pki/admin-key.pem \
    12. --embed-certs=true \
    13. --kubeconfig=/etc/kubernetes/admin.conf
    14.  
    15.  
    16. kubectl config set-context kubernetes-admin@kubernetes \
    17. --cluster=kubernetes \
    18. --user=kubernetes-admin \
    19. --kubeconfig=/etc/kubernetes/admin.conf
    20.  
    21. kubectl config use-context kubernetes-admin@kubernetes \
    22. --kubeconfig=/etc/kubernetes/admin.conf

     kubelet将使用 bootstrap 引导机制，自动颁发证书，所以我们不用配置了。要不然，1万台机器，一个万kubelet，证书配置到明年去。。。

    8、ServiceAccount Key生成

    k8s底层，每创建一个ServiceAccount，都会分配一个Secret，而Secret里面有秘钥，秘钥就是由我们接下来的sa生成的。所以我们提前创建出sa信息

    1. openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
    2.  
    3. openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

    9、发送证书到其他节点

    1. # 在master1上执行
    2. for NODE in k8s-master2 k8s-master3
    3. do
    4. 	for FILE in admin.conf controller-manager.conf scheduler.conf
    5. 	do
    6. 	scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}
    7. 	done
    8. done

    六、高可用配置

    • 高可用配置

      • 如果你不是在创建高可用集群，则无需配置haproxy和keepalived

      • 高可用有很多可选方案

        • nginx

        • haproxy

        • keepalived

        • 云供应商提供的负载均衡产品

    • 云上安装注意事项

      • 云上安装可以直接使用云上的lb，比如阿里云slb，腾讯云elb等

      • 公有云要用公有云自带的负载均衡，比如阿里云的SLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的。

      • 阿里云的话，kubectl控制端不能放在master节点，推荐使用腾讯云，因为阿里云的slb有回环的问题，也就是slb代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题。

    • 青云使用

      • 创建负载均衡器，指定ip地址为我们之前的预留地址

      • 进入负载均衡器，创建监听器

      • 选择TCP，6443端口

      • 添加后端服务器地址与端口

     七、组件启动

    1、所有master执行

    1. mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes
    2.  
    3.  
    4. #三个master节点kube-xx相关的程序都在 /usr/local/bin
    5. for NODE in k8s-master2 k8s-master3
    6. do
    7. 	scp -r /etc/kubernetes/* root@$NODE:/etc/kubernetes/
    8. done

    接下来把master1生成的所有证书全部发给master2,master3

    2、配置apiserver服务

    2.1、配置

     所有Master节点创建kube-apiserver.service

    注意，如果不是高可用集群，192.168.0.250改为master01的地址

    以下文档使用的k8s service网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod网段的重复

    特别注意：docker的网桥默认为 172.17.0.1/16。不要使用这个网段

    1. # 每个master节点都需要执行以下内容
    2. # --advertise-address： 需要改为本master节点的ip
    3. # --service-cluster-ip-range=10.96.0.0/16： 需要改为自己规划的service网段
    4. # --etcd-servers： 改为自己etcd-server的所有地址
    5.  
    6. vi /usr/lib/systemd/system/kube-apiserver.service
    7.  
    8. [Unit]
    9. Description=Kubernetes API Server
    10. Documentation=https://github.com/kubernetes/kubernetes
    11. After=network.target
    12.  
    13. [Service]
    14. ExecStart=/usr/local/bin/kube-apiserver \
    15.       --v=2  \
    16.       --logtostderr=true  \
    17.       --allow-privileged=true  \
    18.       --bind-address=0.0.0.0  \
    19.       --secure-port=6443  \
    20.       --insecure-port=0  \
    21.       --advertise-address=192.168.0.10 \
    22.       --service-cluster-ip-range=10.96.0.0/16  \
    23.       --service-node-port-range=30000-32767  \
    24.       --etcd-servers=https://192.168.0.10:2379,https://192.168.0.11:2379,https://192.168.0.12:2379 \
    25.       --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem  \
    26.       --etcd-certfile=/etc/kubernetes/pki/etcd/etcd.pem  \
    27.       --etcd-keyfile=/etc/kubernetes/pki/etcd/etcd-key.pem  \
    28.       --client-ca-file=/etc/kubernetes/pki/ca.pem  \
    29.       --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
    30.       --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
    31.       --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
    32.       --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
    33.       --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
    34.       --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
    35.       --service-account-issuer=https://kubernetes.default.svc.cluster.local \
    36.       --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
    37.       --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
    38.       --authorization-mode=Node,RBAC  \
    39.       --enable-bootstrap-token-auth=true  \
    40.       --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
    41.       --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
    42.       --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
    43.       --requestheader-allowed-names=aggregator,front-proxy-client  \
    44.       --requestheader-group-headers=X-Remote-Group  \
    45.       --requestheader-extra-headers-prefix=X-Remote-Extra-  \
    46.       --requestheader-username-headers=X-Remote-User
    47.       # --token-auth-file=/etc/kubernetes/token.csv
    48.  
    49. Restart=on-failure
    50. RestartSec=10s
    51. LimitNOFILE=65535
    52.  
    53. [Install]
    54. WantedBy=multi-user.target

    2.2、启动apiserver服务

    1. systemctl daemon-reload && systemctl enable --now kube-apiserver
    2.  
    3.  
    4. #查看状态
    5. systemctl status kube-apiserver

    3、配置controller-manager服务

    3.1、配置

    所有Master节点配置kube-controller-manager.service

    文档使用的k8s Pod网段为196.16.0.0/16，该网段不能和宿主机的网段、k8s Service网段的重复，请按需修改;

    特别注意：docker的网桥默认为 172.17.0.1/16。不要使用这个网

    1. # 所有节点执行
    2. vi /usr/lib/systemd/system/kube-controller-manager.service
    3.  
    4. ## --cluster-cidr=196.16.0.0/16 ： 为Pod的网段。修改成自己想规划的网段
    5.  
    6. [Unit]
    7. Description=Kubernetes Controller Manager
    8. Documentation=https://github.com/kubernetes/kubernetes
    9. After=network.target
    10.  
    11. [Service]
    12. ExecStart=/usr/local/bin/kube-controller-manager \
    13.       --v=2 \
    14.       --logtostderr=true \
    15.       --address=127.0.0.1 \
    16.       --root-ca-file=/etc/kubernetes/pki/ca.pem \
    17.       --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
    18.       --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
    19.       --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
    20.       --kubeconfig=/etc/kubernetes/controller-manager.conf \
    21.       --leader-elect=true \
    22.       --use-service-account-credentials=true \
    23.       --node-monitor-grace-period=40s \
    24.       --node-monitor-period=5s \
    25.       --pod-eviction-timeout=2m0s \
    26.       --controllers=*,bootstrapsigner,tokencleaner \
    27.       --allocate-node-cidrs=true \
    28.       --cluster-cidr=196.16.0.0/16 \
    29.       --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
    30.       --node-cidr-mask-size=24
    31.       
    32. Restart=always
    33. RestartSec=10s
    34.  
    35. [Install]
    36. WantedBy=multi-user.target

    3.2、启动

    1. # 所有master节点执行
    2. systemctl daemon-reload
    3.  
    4. systemctl daemon-reload && systemctl enable --now kube-controller-manager
    5.  
    6. systemctl status kube-controller-manager

    4、配置scheduler

    4.1、配置

    所有Master节点配置kube-scheduler.service

    1. vi /usr/lib/systemd/system/kube-scheduler.service 
    2.  
    3.  
    4.  
    5. [Unit]
    6. Description=Kubernetes Scheduler
    7. Documentation=https://github.com/kubernetes/kubernetes
    8. After=network.target
    9.  
    10. [Service]
    11. ExecStart=/usr/local/bin/kube-scheduler \
    12.       --v=2 \
    13.       --logtostderr=true \
    14.       --address=127.0.0.1 \
    15.       --leader-elect=true \
    16.       --kubeconfig=/etc/kubernetes/scheduler.conf
    17.  
    18. Restart=always
    19. RestartSec=10s
    20.  
    21. [Install]
    22. WantedBy=multi-user.target

    4.2、启动

    1. systemctl daemon-reload
    2.  
    3. systemctl daemon-reload && systemctl enable --now kube-scheduler
    4.  
    5. systemctl status kube-scheduler

    八、TLS与引导启动原理

    1、master1配置bootstrap

    注意，如果不是高可用集群，192.168.0.250:6443改为master1的地址，6443为apiserver的默认端口

    1. #准备一个随机token。但是我们只需要16个字符
    2. head -c 16 /dev/urandom | od -An -t x | tr -d ' '
    3. # 值如下： 737b177d9823531a433e368fcdb16f5f
    4.  
    5. # 生成16个字符的
    6. head -c 8 /dev/urandom | od -An -t x | tr -d ' '
    7. # d683399b7a553977
    1. #设置集群
    2. kubectl config set-cluster kubernetes \
    3. --certificate-authority=/etc/kubernetes/pki/ca.pem \
    4. --embed-certs=true \
    5. --server=https://192.168.0.250:6443 \
    6. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
    7.  
    8. #设置秘钥
    9. kubectl config set-credentials tls-bootstrap-token-user \
    10. --token=l6fy8c.d683399b7a553977 \
    11. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf 
    12.  
    13. #设置上下文
    14. kubectl config set-context tls-bootstrap-token-user@kubernetes \
    15. --cluster=kubernetes \
    16. --user=tls-bootstrap-token-user \
    17. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
    18.  
    19. #使用设置
    20. kubectl config use-context tls-bootstrap-token-user@kubernetes \
    21. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf

    2、master1设置kubectl执行权限

    kubectl 能不能操作集群是看 /root/.kube 下有没有config文件，而config就是我们之前生成的admin.conf，具有操作权限的

    1. # 只在master1生成，因为生产集群，我们只能让一台机器具有操作集群的权限，这样好控制
    2.  
    3. mkdir -p /root/.kube ;
    4. cp /etc/kubernetes/admin.conf /root/.kube/config
    1. #验证
    2. kubectl get nodes
    3.  
    4. # 应该在网络里面开放负载均衡器的6443端口;默认应该不要配置的
    5. [root@k8s-master1 ~]# kubectl get nodes
    6. No resources found
    7. #说明已经可以连接apiserver并获取资源

    3、创建集群引导权限文件

    1. # master准备这个文件 
    2. vi  /etc/kubernetes/bootstrap.secret.yaml
    3.  
    4.  
    5.  
    6. apiVersion: v1
    7. kind: Secret
    8. metadata:
    9.   name: bootstrap-token-l6fy8c
    10.   namespace: kube-system
    11. type: bootstrap.kubernetes.io/token
    12. stringData:
    13.   description: "The default bootstrap token generated by 'kubelet '."
    14.   token-id: l6fy8c
    15.   token-secret: d683399b7a553977
    16.   usage-bootstrap-authentication: "true"
    17.   usage-bootstrap-signing: "true"
    18.   auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
    19.  
    20. ---
    21. apiVersion: rbac.authorization.k8s.io/v1
    22. kind: ClusterRoleBinding
    23. metadata:
    24.   name: kubelet-bootstrap
    25. roleRef:
    26.   apiGroup: rbac.authorization.k8s.io
    27.   kind: ClusterRole
    28.   name: system:node-bootstrapper
    29. subjects:
    30. - apiGroup: rbac.authorization.k8s.io
    31.   kind: Group
    32.   name: system:bootstrappers:default-node-token
    33. ---
    34. apiVersion: rbac.authorization.k8s.io/v1
    35. kind: ClusterRoleBinding
    36. metadata:
    37.   name: node-autoapprove-bootstrap
    38. roleRef:
    39.   apiGroup: rbac.authorization.k8s.io
    40.   kind: ClusterRole
    41.   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
    42. subjects:
    43. - apiGroup: rbac.authorization.k8s.io
    44.   kind: Group
    45.   name: system:bootstrappers:default-node-token
    46. ---
    47. apiVersion: rbac.authorization.k8s.io/v1
    48. kind: ClusterRoleBinding
    49. metadata:
    50.   name: node-autoapprove-certificate-rotation
    51. roleRef:
    52.   apiGroup: rbac.authorization.k8s.io
    53.   kind: ClusterRole
    54.   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
    55. subjects:
    56. - apiGroup: rbac.authorization.k8s.io
    57.   kind: Group
    58.   name: system:nodes
    59. ---
    60. apiVersion: rbac.authorization.k8s.io/v1
    61. kind: ClusterRole
    62. metadata:
    63.   annotations:
    64.     rbac.authorization.kubernetes.io/autoupdate: "true"
    65.   labels:
    66.     kubernetes.io/bootstrapping: rbac-defaults
    67.   name: system:kube-apiserver-to-kubelet
    68. rules:
    69.   - apiGroups:
    70.       - ""
    71.     resources:
    72.       - nodes/proxy
    73.       - nodes/stats
    74.       - nodes/log
    75.       - nodes/spec
    76.       - nodes/metrics
    77.     verbs:
    78.       - "*"
    79. ---
    80. apiVersion: rbac.authorization.k8s.io/v1
    81. kind: ClusterRoleBinding
    82. metadata:
    83.   name: system:kube-apiserver
    84.   namespace: ""
    85. roleRef:
    86.   apiGroup: rbac.authorization.k8s.io
    87.   kind: ClusterRole
    88.   name: system:kube-apiserver-to-kubelet
    89. subjects:
    90.   - apiGroup: rbac.authorization.k8s.io
    91.     kind: User
    92.     name: kube-apiserver
    1. # 应用此文件资源内容
    2. kubectl create -f /etc/kubernetes/bootstrap.secret.yaml

    九、引导Node节点启动

    所有节点的kubelet需要我们引导启动

    1、发送核心证书到节点

    master1节点把核心证书发送到其他节点

    1. cd /etc/kubernetes/  #查看所有信息
    2.  
    3. #执行复制所有令牌操作
    4.  
    5. for NODE in k8s-master2 k8s-master3 k8s-node1 k8s-node2; do
    6.      ssh $NODE mkdir -p /etc/kubernetes/pki/etcd
    7.      for FILE in ca.pem etcd.pem etcd-key.pem; do
    8.        scp /etc/kubernetes/pki/etcd/$FILE $NODE:/etc/kubernetes/pki/etcd/
    9.      done
    10.      for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.conf; do
    11.        scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
    12.  done
    13.  done

    2、所有节点配置kubelet

    1. # 所有节点创建相关目录
    2. mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/
    3.  
    4. ## 所有node节点必须有 kubelet kube-proxy
    5. for NODE in k8s-master2 k8s-master3 k8s-node3 k8s-node1 k8s-node2; do
    6.      scp -r /etc/kubernetes/* root@$NODE:/etc/kubernetes/
    7.  done

    2.1、创建kubelet.service

    1. #所有节点，配置kubelet服务
    2.  
    3. vi  /usr/lib/systemd/system/kubelet.service
    4.  
    5. [Unit]
    6. Description=Kubernetes Kubelet
    7. Documentation=https://github.com/kubernetes/kubernetes
    8. After=docker.service
    9. Requires=docker.service
    10.  
    11. [Service]
    12. ExecStart=/usr/local/bin/kubelet
    13.  
    14. Restart=always
    15. StartLimitInterval=0
    16. RestartSec=10
    17.  
    18. [Install]
    19. WantedBy=multi-user.target
    1. # 所有节点配置kubelet service配置文件
    2. vi /etc/systemd/system/kubelet.service.d/10-kubelet.conf
    3.  
    4. [Service]
    5. Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
    6. Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
    7. Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.4.1"
    8. Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
    9. ExecStart=
    10. ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

    2.2、创建kubelet-conf.yml文件

    1. #所有节点，配置kubelet-conf文件
    2. vi /etc/kubernetes/kubelet-conf.yml
    3. # clusterDNS 为service网络的第10个ip值,改成自己的。如：10.96.0.10
    1. apiVersion: kubelet.config.k8s.io/v1beta1
    2. kind: KubeletConfiguration
    3. address: 0.0.0.0
    4. port: 10250
    5. readOnlyPort: 10255
    6. authentication:
    7.   anonymous:
    8.     enabled: false
    9.   webhook:
    10.     cacheTTL: 2m0s
    11.     enabled: true
    12.   x509:
    13.     clientCAFile: /etc/kubernetes/pki/ca.pem
    14. authorization:
    15.   mode: Webhook
    16.   webhook:
    17.     cacheAuthorizedTTL: 5m0s
    18.     cacheUnauthorizedTTL: 30s
    19. cgroupDriver: systemd
    20. cgroupsPerQOS: true
    21. clusterDNS:
    22. - 10.96.0.10
    23. clusterDomain: cluster.local
    24. containerLogMaxFiles: 5
    25. containerLogMaxSize: 10Mi
    26. contentType: application/vnd.kubernetes.protobuf
    27. cpuCFSQuota: true
    28. cpuManagerPolicy: none
    29. cpuManagerReconcilePeriod: 10s
    30. enableControllerAttachDetach: true
    31. enableDebuggingHandlers: true
    32. enforceNodeAllocatable:
    33. - pods
    34. eventBurst: 10
    35. eventRecordQPS: 5
    36. evictionHard:
    37.   imagefs.available: 15%
    38.   memory.available: 100Mi
    39.   nodefs.available: 10%
    40.   nodefs.inodesFree: 5%
    41. evictionPressureTransitionPeriod: 5m0s  #缩小相应的配置
    42. failSwapOn: true
    43. fileCheckFrequency: 20s
    44. hairpinMode: promiscuous-bridge
    45. healthzBindAddress: 127.0.0.1
    46. healthzPort: 10248
    47. httpCheckFrequency: 20s
    48. imageGCHighThresholdPercent: 85
    49. imageGCLowThresholdPercent: 80
    50. imageMinimumGCAge: 2m0s
    51. iptablesDropBit: 15
    52. iptablesMasqueradeBit: 14
    53. kubeAPIBurst: 10
    54. kubeAPIQPS: 5
    55. makeIPTablesUtilChains: true
    56. maxOpenFiles: 1000000
    57. maxPods: 110
    58. nodeStatusUpdateFrequency: 10s
    59. oomScoreAdj: -999
    60. podPidsLimit: -1
    61. registryBurst: 10
    62. registryPullQPS: 5
    63. resolvConf: /etc/resolv.conf
    64. rotateCertificates: true
    65. runtimeRequestTimeout: 2m0s
    66. serializeImagePulls: true
    67. staticPodPath: /etc/kubernetes/manifests
    68. streamingConnectionIdleTimeout: 4h0m0s
    69. syncFrequency: 1m0s
    70. volumeStatsAggPeriod: 1m0s

     2.3、所有节点启动kubelet

    1. systemctl daemon-reload && systemctl enable --now kubelet
    2.  
    3.  
    4. systemctl status kubelet

    会提示 "Unable to update cni config"。

    接下来配置cni网络即可

    3、kube-proxy配置

    注意，如果不是高可用集群，192.168.0.250:6443改为master1的地址，6443改为apiserver的默认端口

    3.1、生成kube-proxy.conf

    以下操作在master1执行

    1. #创建kube-proxy的sa
    2. kubectl -n kube-system create serviceaccount kube-proxy
    3.  
    4. #创建角色绑定
    5. kubectl create clusterrolebinding system:kube-proxy \
    6. --clusterrole system:node-proxier \
    7. --serviceaccount kube-system:kube-proxy
    8.  
    9. #导出变量，方便后面使用
    10. SECRET=$(kubectl -n kube-system get sa/kube-proxy --output=jsonpath='{.secrets[0].name}')
    11. JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET --output=jsonpath='{.data.token}' | base64 -d)
    12. PKI_DIR=/etc/kubernetes/pki
    13. K8S_DIR=/etc/kubernetes
    14.  
    15. # 生成kube-proxy配置
    16. # --server: 指定自己的apiserver地址或者lb地址
    17. kubectl config set-cluster kubernetes \
    18. --certificate-authority=/etc/kubernetes/pki/ca.pem \
    19. --embed-certs=true \
    20. --server=https://192.168.0.250:6443 \
    21. --kubeconfig=${K8S_DIR}/kube-proxy.conf
    22.  
    23. # kube-proxy秘钥设置
    24. kubectl config set-credentials kubernetes \
    25. --token=${JWT_TOKEN} \
    26. --kubeconfig=/etc/kubernetes/kube-proxy.conf
    27.  
    28.  
    29. kubectl config set-context kubernetes \
    30. --cluster=kubernetes \
    31. --user=kubernetes \
    32. --kubeconfig=/etc/kubernetes/kube-proxy.conf
    33.  
    34.  
    35. kubectl config use-context kubernetes \
    36. --kubeconfig=/etc/kubernetes/kube-proxy.conf
    1. #把生成的 kube-proxy.conf 传给每个节点
    2. for NODE in k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3; do
    3.       scp /etc/kubernetes/kube-proxy.conf $NODE:/etc/kubernetes/
    4.  done

    3.2、配置kube-proxy.service

    1. # 所有节点配置 kube-proxy.service 服务，一会儿设置为开机启动
    2. vi /usr/lib/systemd/system/kube-proxy.service
    3.  
    4. [Unit]
    5. Description=Kubernetes Kube Proxy
    6. Documentation=https://github.com/kubernetes/kubernetes
    7. After=network.target
    8.  
    9. [Service]
    10. ExecStart=/usr/local/bin/kube-proxy \
    11.   --config=/etc/kubernetes/kube-proxy.yaml \
    12.   --v=2
    13.  
    14. Restart=always
    15. RestartSec=10s
    16.  
    17. [Install]
    18. WantedBy=multi-user.target

    3.3、准备kube-proxy.yaml

    一定注意修改自己的Pod网段范围

    1. # 所有机器执行
    2. vi /etc/kubernetes/kube-proxy.yaml
    1. apiVersion: kubeproxy.config.k8s.io/v1alpha1
    2. bindAddress: 0.0.0.0
    3. clientConnection:
    4.   acceptContentTypes: ""
    5.   burst: 10
    6.   contentType: application/vnd.kubernetes.protobuf
    7.   kubeconfig: /etc/kubernetes/kube-proxy.conf   #kube-proxy引导文件
    8.   qps: 5
    9. clusterCIDR: 196.16.0.0/16  #修改为自己的Pod-CIDR
    10. configSyncPeriod: 15m0s
    11. conntrack:
    12.   max: null
    13.   maxPerCore: 32768
    14.   min: 131072
    15.   tcpCloseWaitTimeout: 1h0m0s
    16.   tcpEstablishedTimeout: 24h0m0s
    17. enableProfiling: false
    18. healthzBindAddress: 0.0.0.0:10256
    19. hostnameOverride: ""
    20. iptables:
    21.   masqueradeAll: false
    22.   masqueradeBit: 14
    23.   minSyncPeriod: 0s
    24.   syncPeriod: 30s
    25. ipvs:
    26.   masqueradeAll: true
    27.   minSyncPeriod: 5s
    28.   scheduler: "rr"
    29.   syncPeriod: 30s
    30. kind: KubeProxyConfiguration
    31. metricsBindAddress: 127.0.0.1:10249
    32. mode: "ipvs"
    33. nodePortAddresses: null
    34. oomScoreAdj: -999
    35. portRange: ""
    36. udpIdleTimeout: 250ms

    3.4、启动kube-proxy

    所有节点启动

    1. systemctl daemon-reload && systemctl enable --now kube-proxy
    2. systemctl status kube-proxy

     十、部署calico

    可以参照calico私有云部署指南

    1. # 下载官网calico
    2. curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico.yaml
    3. ## 把这个镜像修改成国内镜像
    4.  
    5.  
    6. # 修改一些我们自定义的. 修改etcd集群地址
    7. sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.0.10:2379,https://192.168.0.11:2379,https://192.168.0.12:2379"#g' calico.yaml
    8.  
    9.  
    10. # etcd的证书内容，需要base64编码设置到yaml中
    11. ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.pem | base64 -w 0 `
    12. ETCD_CERT=`cat /etc/kubernetes/pki/etcd/etcd.pem | base64 -w 0 `
    13. ETCD_KEY=`cat /etc/kubernetes/pki/etcd/etcd-key.pem | base64 -w 0 `
    14.  
    15. # 替换etcd中的证书base64编码后的内容
    16. sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico.yaml
    17.  
    18.  
    19. #打开 etcd_ca 等默认设置（calico启动后自己生成）。
    20. sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico.yaml
    21.  
    22. # 修改自己的Pod网段 196.16.0.0/16
    23. POD_SUBNET="196.16.0.0/16"
    24. sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico.yaml
    25. # 一定确定自己是否修改好了
    26.  
    27. #确认calico是否修改好
    28. grep "CALICO_IPV4POOL_CIDR" calico.yaml -A 1

    1. # 应用calico配置
    2. kubectl apply -f calico.yaml

    十一、部署coreDNS

    1. git clone https://github.com/coredns/deployment.git
    2. cd deployment/kubernetes
    3.  
    4. #10.96.0.10 改为 service 网段的 第 10 个ip
    5. ./deploy.sh -s -i 10.96.0.10 | kubectl apply -f -

    十二、给机器打上role标签

    1. kubectl label node k8s-master1 node-role.kubernetes.io/master=''
    2. kubectl label node k8s-master2 node-role.kubernetes.io/master=''
    3. kubectl label node k8s-master3 node-role.kubernetes.io/master=''
    4.  
    5. kubectl taints node k8s-master1

    十三、集群验证

    • 验证Pod网络可访问性

      • 同名称空间，不同名称空间可以使用 ip 互相访问

      • 跨机器部署的Pod也可以互相访问

    • 验证Service网络可访问性

      • 集群机器使用serviceIp可以负载均衡访问

      • pod内部可以访问service域名 serviceName.namespace

      • pod可以访问跨名称空间的service

    1. # 部署以下内容进行测试
    2.  
    3. apiVersion: apps/v1
    4. kind: Deployment
    5. metadata:
    6.   name:  nginx-01
    7.   namespace: default
    8.   labels:
    9.     app:  nginx-01
    10. spec:
    11.   selector:
    12.     matchLabels:
    13.       app: nginx-01
    14.   replicas: 1
    15.   template:
    16.     metadata:
    17.       labels:
    18.         app:  nginx-01
    19.     spec:
    20.       containers:
    21.       - name:  nginx-01
    22.         image:  nginx
    23. ---
    24. apiVersion: v1
    25. kind: Service
    26. metadata:
    27.   name: nginx-svc
    28.   namespace: default
    29. spec:
    30.   selector:
    31.     app:  nginx-01
    32.   type: ClusterIP
    33.   ports:
    34.   - name: nginx-svc
    35.     port: 80
    36.     targetPort: 80
    37.     protocol: TCP
    38. ---
    39. apiVersion: v1
    40. kind: Namespace
    41. metadata:
    42.   name: hello
    43. spec: {}
    44. ---
    45. apiVersion: apps/v1
    46. kind: Deployment
    47. metadata:
    48.   name:  nginx-hello
    49.   namespace: hello
    50.   labels:
    51.     app:  nginx-hello
    52. spec:
    53.   selector:
    54.     matchLabels:
    55.       app: nginx-hello
    56.   replicas: 1
    57.   template:
    58.     metadata:
    59.       labels:
    60.         app:  nginx-hello
    61.     spec:
    62.       containers:
    63.       - name:  nginx-hello
    64.         image:  nginx
    65. ---
    66. apiVersion: v1
    67. kind: Service
    68. metadata:
    69.   name: nginx-svc-hello
    70.   namespace: hello
    71. spec:
    72.   selector:
    73.     app:  nginx-hello
    74.   type: ClusterIP
    75.   ports:
    76.   - name: nginx-svc-hello
    77.     port: 80
    78.     targetPort: 80
    79.     protocol: TCP
    1. # 给两个master标识为worker
    2. kubectl label node k8s-node3 node-role.kubernetes.io/worker=''
    3. kubectl label node k8s-master3 node-role.kubernetes.io/worker=''
    4. kubectl label node k8s-node1 node-role.kubernetes.io/worker=''
    5. kubectl label node k8s-node2 node-role.kubernetes.io/worker=''
    6.  
    7. # 给master1打上污点。二进制部署的集群，默认master是没有污点的，可以任意调度。我们最好给一个master打上污点，保证master最小可用
    8. kubectl label node k8s-master3 node-role.kubernetes.io/master=''
    9. kubectl taint nodes k8s-master1 node-role.kubernetes.io/master=:NoSchedule

    文末惊喜 

    开发云特价优惠

    【开发云】年年都是折扣价，不用四处薅羊毛

    • 📢博客主页：https://lansonli.blog.csdn.net
    • 📢欢迎点赞 👍 收藏 ⭐留言 📝 如有错误敬请指正！
    • 📢本文由 Lansonli 原创，首发于 CSDN博客🙉
    • 📢停下休息的时候不要忘了别人还在奔跑，希望大家抓紧时间学习，全力奔赴更美好的生活✨
  • 相关阅读:
    【C++】STL — string的使用 + 模拟实现
    ros2_control【B站WMGIII教学学习记录】1
    全志A33使用主线U-Boot
    透视金融科技Q3财报：规模效应显现，深化小微服务覆盖面
    spring基本使用
    正射影像矫正--基于无人机图片
    软件测试/测试开发丨学会与 AI 对话，高效提升学习效率
    Ikigai: 享受生命的意义
    SpringCloudGateway 入门
    sketch for Mac快捷键大全
  • 原文地址：https://blog.csdn.net/xiaoweite1/article/details/125380832