• Kubernetes IPVS和IPTABLES

    Kubernetes IPVS和IPTABLES

    什么是IPVS

    IPVS（IP Virtual Server，IP虚拟服务器）实现了传输层的负载平衡，通常称为4 LAN（四层局域网）交换，是Linux内核的一部分。

    IPVS在主机上运行，在真实服务器集群前面充当负载平衡器。IPVS可以将基于 TCP 和 UDP 的服务请求定向到真实服务器上。

    IPVS vs IPTABLES

    IPVS 模式在 Kubernetes v1.8中引入，在v1.9中成为测试版，在v1.11中成为GA。
    IPTABLES模式是在v1.1版本中加入的，从v1.2版本开始成为默认的操作模式。

    IPVS和IPTABLES都是基于netfilter的。IPVS模式和IPTABLES模式的区别如下：

    1. IPVS为大型集群提供更好的可扩展性和性能。

    2. IPVS比IPTABLES支持更复杂的负载平衡算法（最小负载、最小连接、定位、加权等）。

    3. IPVS 支持服务器健康检查和连接重试等。

    IPVS 对 IPTABLES 的依赖

    IPVS代理使用IPTABLES做数据包过滤、SNAT或伪装。具体来说，IPVS代理将使用ipset来存储需要DROP或做伪装的流量的源地址或目的地址，以确保无论我们有多少服务，IPTABLES规则的数量不变。

    下面是IPVS代理服务器使用的ipset集的表格。

    set namemembersusage
    KUBE-CLUSTER-IPAll service IP + portMark-Masq for cases that masquerade-all=true or clusterCIDRspecified
    KUBE-LOOP-BACKAll service IP + port + IPmasquerade for solving hairpin purpose
    KUBE-EXTERNAL-IPservice external IP + portmasquerade for packages to external IPs
    KUBE-LOAD-BALANCERload balancer ingress IP + portmasquerade for packages to load balancer type service
    KUBE-LOAD-BALANCER-LOCALLB ingress IP + port with externalTrafficPolicy=localaccept packages to load balancer with externalTrafficPolicy=local
    KUBE-LOAD-BALANCER-FWload balancer ingress IP + port with loadBalancerSourceRangespackage filter for load balancer with loadBalancerSourceRangesspecified
    KUBE-LOAD-BALANCER-SOURCE-CIDRload balancer ingress IP + port + source CIDRpackage filter for load balancer with loadBalancerSourceRangesspecified
    KUBE-NODE-PORT-TCPnodeport type service TCP portmasquerade for packets to nodePort(TCP)
    KUBE-NODE-PORT-LOCAL-TCPnodeport type service TCP port with externalTrafficPolicy=localaccept packages to nodeport service with externalTrafficPolicy=local
    KUBE-NODE-PORT-UDPnodeport type service UDP portmasquerade for packets to nodePort(UDP)
    KUBE-NODE-PORT-LOCAL-UDPnodeport type service UDP port with externalTrafficPolicy=localaccept packages to nodeport service with externalTrafficPolicy=local

    在以下情况下，IPVS 代理将依赖 IPTABLES。

    1. kube-proxy以–masquerade-all=true启动

    如果kube-proxy以--masquerade-all=true启动，IPVS代理将伪装所有访问服务集群IP的流量，这与IPTABLES代理的行为相同。假设kube-proxy指定了标志--masquerade-all=true，那么IPVS代理安装的IPTABLES应该如下所示：

    1. # iptables -t nat -nL
    2.  
    3. Chain PREROUTING (policy ACCEPT)
    4. target     prot opt source               destination
    5. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    6.  
    7. Chain OUTPUT (policy ACCEPT)
    8. target     prot opt source               destination
    9. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    10.  
    11. Chain POSTROUTING (policy ACCEPT)
    12. target     prot opt source               destination
    13. KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    14.  
    15. Chain KUBE-MARK-MASQ (2 references)
    16. target     prot opt source               destination
    17. MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000
    18.  
    19. Chain KUBE-POSTROUTING (1 references)
    20. target     prot opt source               destination
    21. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
    22. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src
    23.  
    24. Chain KUBE-SERVICES (2 references)
    25. target     prot opt source               destination
    26. KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst
    27. ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst

    2. 在kube-proxy启动时指定集群CIDR

    如果kube-proxy以--cluster-cidr=启动，IPVS代理将伪装访问服务集群IP的非集群流量，其行为与IPTABLES代理相同。假设kube-proxy提供的集群cidr是10.244.16.0/24，那么IPVS代理安装的IPTABLES应该如下所示。

    1. # iptables -t nat -nL
    2.  
    3. Chain PREROUTING (policy ACCEPT)
    4. target     prot opt source               destination
    5. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    6.  
    7. Chain OUTPUT (policy ACCEPT)
    8. target     prot opt source               destination
    9. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    10.  
    11. Chain POSTROUTING (policy ACCEPT)
    12. target     prot opt source               destination
    13. KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    14.  
    15. Chain KUBE-MARK-MASQ (3 references)
    16. target     prot opt source               destination
    17. MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000
    18.  
    19. Chain KUBE-POSTROUTING (1 references)
    20. target     prot opt source               destination
    21. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
    22. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src
    23.  
    24. Chain KUBE-SERVICES (2 references)
    25. target     prot opt source               destination
    26. KUBE-MARK-MASQ  all  -- !10.244.16.0/24       0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst
    27. ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst

    3. loadBalancer类型的服务

    对于loadBalancer类型的服务，IPVS代理将安装IPTABLES与ipset KUBE-LOAD-BALANCER匹配。特别是当服务的LoadBalancerSourceRanges被指定或指定externalTrafficPolicy=local时，IPVS代理将创建ipset集KUBE-LOAD-BALANCER-LOCAL/KUBE-LOAD-BALANCER-FW/KUBE-LOAD-BALANCER-SOURCE-CIDR并相应地安装IPTABLES，它应该看起来像下面所示。

    1. # iptables -t nat -nL
    2.  
    3. Chain PREROUTING (policy ACCEPT)
    4. target     prot opt source               destination
    5. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    6.  
    7. Chain OUTPUT (policy ACCEPT)
    8. target     prot opt source               destination
    9. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    10.  
    11. Chain POSTROUTING (policy ACCEPT)
    12. target     prot opt source               destination
    13. KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    14.  
    15. Chain KUBE-FIREWALL (1 references)
    16. target     prot opt source               destination
    17. RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER-SOURCE-CIDR dst,dst,src
    18. KUBE-MARK-DROP  all  --  0.0.0.0/0            0.0.0.0/0
    19.  
    20. Chain KUBE-LOAD-BALANCER (1 references)
    21. target     prot opt source               destination
    22. KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER-FW dst,dst
    23. RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER-LOCAL dst,dst
    24. KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0
    25.  
    26. Chain KUBE-MARK-DROP (1 references)
    27. target     prot opt source               destination
    28. MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000
    29.  
    30. Chain KUBE-MARK-MASQ (2 references)
    31. target     prot opt source               destination
    32. MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000
    33.  
    34. Chain KUBE-POSTROUTING (1 references)
    35. target     prot opt source               destination
    36. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
    37. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src
    38.  
    39. Chain KUBE-SERVICES (2 references)
    40. target     prot opt source               destination
    41. KUBE-LOAD-BALANCER  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER dst,dst
    42. ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER dst,dst

    4. NodePort类型的服务

    对于NodePort类型的服务，IPVS代理将安装IPTABLES与ipset KUBE-NODE-PORT-TCP/KUBE-NODE-PORT-UDP的匹配。当指定externalTrafficPolicy=local时，IPVS代理将创建ipset集KUBE-NODE-PORT-LOCAL-TCP/KUBE-NODE-PORT-LOCAL-UDP并相应地安装IPTABLES，这应该是如下所示的：

    假设服务的TCP类型为nodePort。

    1. Chain PREROUTING (policy ACCEPT)
    2. target     prot opt source               destination
    3. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    4.  
    5. Chain OUTPUT (policy ACCEPT)
    6. target     prot opt source               destination
    7. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    8.  
    9. Chain POSTROUTING (policy ACCEPT)
    10. target     prot opt source               destination
    11. KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    12.  
    13. Chain KUBE-MARK-MASQ (2 references)
    14. target     prot opt source               destination
    15. MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000
    16.  
    17. Chain KUBE-NODE-PORT (1 references)
    18. target     prot opt source               destination
    19. RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-NODE-PORT-LOCAL-TCP dst
    20. KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0
    21.  
    22. Chain KUBE-POSTROUTING (1 references)
    23. target     prot opt source               destination
    24. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
    25. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src
    26.  
    27. Chain KUBE-SERVICES (2 references)
    28. target     prot opt source               destination
    29. KUBE-NODE-PORT  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-NODE-PORT-TCP dst

    5. 指定externalIP的服务

    对于指定了外部IP的服务，IPVS代理将安装IPTABLES与ipset KUBE-EXTERNAL-IP匹配，假设我们有指定了外部IP的服务，IPTABLES规则应该如下所示：

    1. Chain PREROUTING (policy ACCEPT)
    2. target     prot opt source               destination
    3. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    4.  
    5. Chain OUTPUT (policy ACCEPT)
    6. target     prot opt source               destination
    7. KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    8.  
    9. Chain POSTROUTING (policy ACCEPT)
    10. target     prot opt source               destination
    11. KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    12.  
    13. Chain KUBE-MARK-MASQ (2 references)
    14. target     prot opt source               destination
    15. MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000
    16.  
    17. Chain KUBE-POSTROUTING (1 references)
    18. target     prot opt source               destination
    19. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
    20. MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src
    21.  
    22. Chain KUBE-SERVICES (2 references)
    23. target     prot opt source               destination
    24. KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-EXTERNAL-IP dst,dst
    25. ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-EXTERNAL-IP dst,dst PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    26. ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-EXTERNAL-IP dst,dst ADDRTYPE match dst-type LOCAL

    Kubernetes使用IPVS

    安装IPVS

    CentOS

    yum install ipset ipvsadm -y

    Ubuntu

    1. apt-get install ipset ipvsadm -y
    2.

    设置

    1. cat > /etc/sysconfig/modules/ipvs.modules <<EOF
    2. #!/bin/bash
    3. modprobe -- ip_vs
    4. modprobe -- ip_vs_rr
    5. modprobe -- ip_vs_wrr
    6. modprobe -- ip_vs_sh
    7. modprobe -- nf_conntrack_ipv4
    8. EOF
    9. chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
  • 相关阅读:
    如何使用并查集解决朋友圈问题？
    在Ubuntu中使用Cairosvg将svg批量转换为png
    Eureka Client启动后就关闭 Unregistering application xxx with eureka with status DOWN
    Mac，Windows11，Windows10局域网互传共享文件
    C# 海康威视平台API接入 和网页摄像头部署
    Javascript 事件的动态绑定
    Spring Cloud Alibaba【授权规则、系统自适应限流、SentinelResource注解配置详解之只配 置fallback】（八）
    pytorch初学笔记（十三）：神经网络基本结构之Sequential层的使用以及搭建完整的小型神经网络实战
    地下管线探测仪 光电缆路由探测仪 管线仪 定位仪 探测0-6米 操作简捷 高精度
    2022.11.17 HDU-4911 Inversion
  • 原文地址：https://blog.csdn.net/sinat_40572875/article/details/127932730