【k8s部署elasticsearch】k8s环境下安装elasticsearch集群和kibana

简介

k8s集群中搭建有elasticsearch服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将elasticsearch集群的搭建采用hostpath+亲和性的权重+多个副本分区的方式来实现数据持久化和高可用。

一.条件及环境说明

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator。

二.需求说明

• 搭建三个节点的elasticsearch集群，并配置三个主分区，2个副本分区。
• 高可用：有一个k8s节点长期故障后或重启pod之后也不影响使用。
• 高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
• 数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
• 安装简单、管理维护容易

三.实现原理及说明

四.详细步骤

4.1.规划节点标签

注：这里的角色划分是指：redis由于是一主两从，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就切换到备节点，例如：名称为autopocket-es-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建三个statefulset和service headless配置

1.elasticsearch_a.yaml 节点a的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autopocket-es-a
  namespace: zx-app
  labels:
    app: autopocket-es-a 
    appname: pcauto-pocket
    appgroup: autopocket-es
spec:
  serviceName: "autopocket-es-a"
  replicas: 1 
  selector:
    matchLabels:
      app: autopocket-es-a
      appgroup: autopocket-es
  template:
    metadata:
      labels:
        app: autopocket-es-a
        appgroup: autopocket-es
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-a-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-a-slave
      containers:
      - name: elasticsearch
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
        command: ["/bin/sh","-c"]
        args:
        - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
        env:
        - name: cluster.name
          value: autopocket-es 
        - name: K8S_SERVICE_NAME
          value: autopocket-es-a 
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: discovery.seed_hosts
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
        - name: cluster.initial_master_nodes
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
        - name: node.name
          value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: path.data
          value: /home/elastic/$(POD_NAME)/data
        - name: path.logs
          value: /home/elastic/$(POD_NAME)/logs
        - name: xpack.security.enabled
          value: "false"
        ports:
        - name: rest
          containerPort: 9200
        - name: inter-node
          containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /home/elastic
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: elasticsearch-data 
          hostPath:
            path: /data/es_data
            type: ""
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-a
  namespace: zx-app
  labels:
    appname: pcauto-pocket
    app: autopocket-es-a
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
  selector:
    app: autopocket-es-a

2.elasticsearch_b.yaml 节点b的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autopocket-es-b
  namespace: zx-app
  labels:
    app: autopocket-es-b 
    appname: pcauto-pocket
    appgroup: autopocket-es
spec:
  serviceName: "autopocket-es-b"
  replicas: 1 
  selector:
    matchLabels:
      app: autopocket-es-b
      appgroup: autopocket-es
  template:
    metadata:
      labels:
        app: autopocket-es-b
        appgroup: autopocket-es
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-b-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-b-slave
      containers:
      - name: elasticsearch
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
        command: ["/bin/sh","-c"]
        args:
        - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
        env:
        - name: cluster.name
          value: autopocket-es 
        - name: K8S_SERVICE_NAME
          value: autopocket-es-b 
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: discovery.seed_hosts
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
        - name: cluster.initial_master_nodes
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
        - name: node.name
          value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: path.data
          value: /home/elastic/$(POD_NAME)/data
        - name: path.logs
          value: /home/elastic/$(POD_NAME)/logs
        - name: xpack.security.enabled
          value: "false"
        ports:
        - name: rest
          containerPort: 9200
        - name: inter-node
          containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /home/elastic
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: elasticsearch-data 
          hostPath:
            path: /data/es_data
            type: ""
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-b
  namespace: zx-app
  labels:
    appname: pcauto-pocket
    app: autopocket-es-b
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
  selector:
    app: autopocket-es-b

3.elasticsearch_c.yaml 节点c的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autopocket-es-c
  namespace: zx-app
  labels:
    app: autopocket-es-c 
    appname: pcauto-pocket
    appgroup: autopocket-es
spec:
  serviceName: "autopocket-es-c"
  replicas: 1 
  selector:
    matchLabels:
      app: autopocket-es-c
      appgroup: autopocket-es
  template:
    metadata:
      labels:
        app: autopocket-es-c
        appgroup: autopocket-es
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-c-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-c-slave
      containers:
      - name: elasticsearch
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
        command: ["/bin/sh","-c"]
        args:
        - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
        env:
        - name: cluster.name
          value: autopocket-es 
        - name: K8S_SERVICE_NAME
          value: autopocket-es-c 
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: discovery.seed_hosts
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
        - name: cluster.initial_master_nodes
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
        - name: node.name
          value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: path.data
          value: /home/elastic/$(POD_NAME)/data
        - name: path.logs
          value: /home/elastic/$(POD_NAME)/logs
        - name: xpack.security.enabled
          value: "false"
        ports:
        - name: rest
          containerPort: 9200
        - name: inter-node
          containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /home/elastic
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: elasticsearch-data 
          hostPath:
            path: /data/es_data
            type: ""
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-c
  namespace: zx-app
  labels:
    appname: pcauto-pocket
    app: autopocket-es-c
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
  selector:
    app: autopocket-es-c

4.3.创建service配置

给9200 9300配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

apiVersion: v1
kind: Service
metadata:
  name: autopocket-es
  namespace: zx-app
  labels:
    app: autopocket-es
    appname: pcauto-zx
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  selector:
    appgroup: autopocket-es
  type: LoadBalancer 

五.安装kibana

apiVersion: apps/v1
kind: Deployment
metadata:
  name: autopocket-es-kibana
  namespace: zx-app
  labels:
    app: autopocket-es-kibana
    appname: pcauto-pocket
spec:
  replicas: 1
  selector:
    matchLabels:
      app: autopocket-es-kibana
  template:
    metadata:
      labels:
        app: autopocket-es-kibana
    spec:
      containers:
      - name: kibana
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/kibana:7.17.22"
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: '["http://autopocket-es-a-0.autopocket-es-a:9200","http://autopocket-es-b-0.autopocket-es-b:9200","http://autopocket-es-c-0.autopocket-es-c:9200"]'
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-kibana
  namespace: zx-app
  labels:
    app: autopocket-es-kibana
    appname: pcauto-pocket
spec:
  selector:
    app: autopocket-es-kibana
  ports:
  - port: 5601
    targetPort: 5601
  type: LoadBalancer

查看集群健康情况：http://10.16.xx.xx:9200/_cat/health?v

elasticsearch集群状态查看：http://10.16.1xx.xx:9200/_cat/nodes?v

索引状态查看：http://10.16.1xx.xx:9200/_cat/indices?v

kibana地址登录：http://10.16.1xx.xx:5601/app/home#/

六.调整索引分区

调整已存在索引的副本分区：curl -XPUT -H "Content-Type:application/json" -d '{"settings":{"number_of_replicas":2}}' 'http://10.16.xx.xx:9200/testindex/_settings'

调整集群最大分片数：curl -XPUT -H "Content-Type:application/json" http://10.16.xx.xx:9200/_cluster/settings -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}'

调整默认主分片和副本分片数: curl -X PUT -H "Content-Type:application/json" -d '{"index_patterns":["*"],"settings":{"number_of_shards":3,"number_of_replicas":2}}' http://10.16.xx.xx:9200/_template/template_http_request_record

七.安装说明

• 需要先在k8s的节点创建/data/es_data的路径来存放数据。
• 使用配置前将配置中的autopocket替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-pocket，修改成自己的应用名称，该标签也可以删除。
• 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的elasticsearch:7.17.22 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用elasticsearch:7.17.22,kibana的镜像是kibana:7.17.22。否则配置一个可以通的镜像地址。
• 默认没有配置用户名和密码。
• 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。