• Elasticsearch：使用 docker compose 来实现热温冷架构的 Elasticsearch 集群

    在我们的许多练习中，我需要模拟索引生命管理周期的环境。在这样的集群中，我们需要模拟出热（hot），温（warm）及冷（cold）的架构。那么我们该如何进行模拟呢？

    我们参考之前的文章 “Elasticsearch：使用 Docker compose 来一键部署 Elastic Stack 8.x”。我们使用 docker compose 来进行部署。

    首先，我们按照如下的步骤来进行修改。

    .env

    .env 文件设置运行 docker-compose.yml 配置文件时使用的环境变量。 确保使用 ELASTIC_PASSWORD 和 KIBANA_PASSWORD 变量为 elastic 和 kibana_system 用户指定密码。 这些变量由 docker-compose.yml 文件引用。
     

    1. # Password for the 'elastic' user (at least 6 characters)
    2. ELASTIC_PASSWORD=password
    3.  
    4. # Password for the 'kibana_system' user (at least 6 characters)
    5. KIBANA_PASSWORD=password
    6.  
    7. # Version of Elastic products
    8. STACK_VERSION=8.5.1
    9.  
    10. # Set the cluster name
    11. CLUSTER_NAME=docker-cluster
    12.  
    13. # Set to 'basic' or 'trial' to automatically start the 30-day trial
    14. LICENSE=basic
    15. #LICENSE=trial
    16.  
    17. # Port to expose Elasticsearch HTTP API to the host
    18. ES_PORT=9200
    19. #ES_PORT=127.0.0.1:9200
    20.  
    21. # Port to expose Kibana to the host
    22. KIBANA_PORT=5601
    23. #KIBANA_PORT=80
    24.  
    25. # Increase or decrease based on the available host memory (in bytes)
    26. MEM_LIMIT=1073741824
    27.  
    28. # Project namespace (defaults to the current folder name if not set)
    29. #COMPOSE_PROJECT_NAME=myproject

    在上面，我们设定 Elastic Stack 的版本为最新的 8.5.1。在这个文件中，我们同时也设定 elastic 这个超级用户的密码。

    docker-compose.yml

    这个 docker-compose.yml 文件创建了一个启用了身份验证和网络加密的三节点安全 Elasticsearch 集群，以及一个安全连接到它的 Kibana 实例。在我们的设置中，我们分别设置三个节点为 data_hot，data_warm 及 data_cold。有关 data tiers 的介绍，请阅读我的另外一篇文章 “Elastic：Data tiers 介绍及索引生命周期管理 - 7.10 之后版本”。

    修改后的 docker-compose.yml 文件如下：

    docker-compose.yml

    1. version: "2.2"
    2.  
    3. services:
    4.   setup:
    5.     image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    6.     volumes:
    7.       - certs:/usr/share/elasticsearch/config/certs
    8.     user: "0"
    9.     command: >
    10.       bash -c '
    11.         if [ x${ELASTIC_PASSWORD} == x ]; then
    12.           echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
    13.           exit 1;
    14.         elif [ x${KIBANA_PASSWORD} == x ]; then
    15.           echo "Set the KIBANA_PASSWORD environment variable in the .env file";
    16.           exit 1;
    17.         fi;
    18.         if [ ! -f certs/ca.zip ]; then
    19.           echo "Creating CA";
    20.           bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
    21.           unzip config/certs/ca.zip -d config/certs;
    22.         fi;
    23.         if [ ! -f certs/certs.zip ]; then
    24.           echo "Creating certs";
    25.           echo -ne \
    26.           "instances:\n"\
    27.           "  - name: es01\n"\
    28.           "    dns:\n"\
    29.           "      - es01\n"\
    30.           "      - localhost\n"\
    31.           "    ip:\n"\
    32.           "      - 127.0.0.1\n"\
    33.           "  - name: es02\n"\
    34.           "    dns:\n"\
    35.           "      - es02\n"\
    36.           "      - localhost\n"\
    37.           "    ip:\n"\
    38.           "      - 127.0.0.1\n"\
    39.           "  - name: es03\n"\
    40.           "    dns:\n"\
    41.           "      - es03\n"\
    42.           "      - localhost\n"\
    43.           "    ip:\n"\
    44.           "      - 127.0.0.1\n"\
    45.           > config/certs/instances.yml;
    46.           bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
    47.           unzip config/certs/certs.zip -d config/certs;
    48.         fi;
    49.         echo "Setting file permissions"
    50.         chown -R root:root config/certs;
    51.         find . -type d -exec chmod 750 \{\} \;;
    52.         find . -type f -exec chmod 640 \{\} \;;
    53.         echo "Waiting for Elasticsearch availability";
    54.         until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
    55.         echo "Setting kibana_system password";
    56.         until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
    57.         echo "All done!";
    58.       '
    59.     healthcheck:
    60.       test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
    61.       interval: 1s
    62.       timeout: 5s
    63.       retries: 120
    64.  
    65.   es01:
    66.     depends_on:
    67.       setup:
    68.         condition: service_healthy
    69.     image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    70.     volumes:
    71.       - certs:/usr/share/elasticsearch/config/certs
    72.       - esdata01:/usr/share/elasticsearch/data
    73.     ports:
    74.       - ${ES_PORT}:9200
    75.     environment:
    76.       - node.name=es01
    77.       - node.roles=data_hot,data_content,master,ingest
    78.       - cluster.name=${CLUSTER_NAME}
    79.       - cluster.initial_master_nodes=es01,es02,es03
    80.       - discovery.seed_hosts=es02,es03
    81.       - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    82.       - bootstrap.memory_lock=true
    83.       - xpack.security.enabled=true
    84.       - xpack.security.http.ssl.enabled=true
    85.       - xpack.security.http.ssl.key=certs/es01/es01.key
    86.       - xpack.security.http.ssl.certificate=certs/es01/es01.crt
    87.       - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
    88.       - xpack.security.http.ssl.verification_mode=certificate
    89.       - xpack.security.transport.ssl.enabled=true
    90.       - xpack.security.transport.ssl.key=certs/es01/es01.key
    91.       - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
    92.       - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
    93.       - xpack.security.transport.ssl.verification_mode=certificate
    94.       - xpack.license.self_generated.type=${LICENSE}
    95.     mem_limit: ${MEM_LIMIT}
    96.     ulimits:
    97.       memlock:
    98.         soft: -1
    99.         hard: -1
    100.     healthcheck:
    101.       test:
    102.         [
    103.           "CMD-SHELL",
    104.           "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
    105.         ]
    106.       interval: 10s
    107.       timeout: 10s
    108.       retries: 120
    109.  
    110.   es02:
    111.     depends_on:
    112.       - es01
    113.     image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    114.     volumes:
    115.       - certs:/usr/share/elasticsearch/config/certs
    116.       - esdata02:/usr/share/elasticsearch/data
    117.     environment:
    118.       - node.name=es02
    119.       - node.roles=data_warm,data_content,master,ingest
    120.       - cluster.name=${CLUSTER_NAME}
    121.       - cluster.initial_master_nodes=es01,es02,es03
    122.       - discovery.seed_hosts=es01,es03
    123.       - bootstrap.memory_lock=true
    124.       - xpack.security.enabled=true
    125.       - xpack.security.http.ssl.enabled=true
    126.       - xpack.security.http.ssl.key=certs/es02/es02.key
    127.       - xpack.security.http.ssl.certificate=certs/es02/es02.crt
    128.       - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
    129.       - xpack.security.http.ssl.verification_mode=certificate
    130.       - xpack.security.transport.ssl.enabled=true
    131.       - xpack.security.transport.ssl.key=certs/es02/es02.key
    132.       - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
    133.       - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
    134.       - xpack.security.transport.ssl.verification_mode=certificate
    135.       - xpack.license.self_generated.type=${LICENSE}
    136.     mem_limit: ${MEM_LIMIT}
    137.     ulimits:
    138.       memlock:
    139.         soft: -1
    140.         hard: -1
    141.     healthcheck:
    142.       test:
    143.         [
    144.           "CMD-SHELL",
    145.           "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
    146.         ]
    147.       interval: 10s
    148.       timeout: 10s
    149.       retries: 120
    150.  
    151.   es03:
    152.     depends_on:
    153.       - es02
    154.     image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    155.     volumes:
    156.       - certs:/usr/share/elasticsearch/config/certs
    157.       - esdata03:/usr/share/elasticsearch/data
    158.     environment:
    159.       - node.name=es03
    160.       - node.roles=data_cold,data_content,master,ingest
    161.       - cluster.name=${CLUSTER_NAME}
    162.       - cluster.initial_master_nodes=es01,es02,es03
    163.       - discovery.seed_hosts=es01,es02
    164.       - bootstrap.memory_lock=true
    165.       - xpack.security.enabled=true
    166.       - xpack.security.http.ssl.enabled=true
    167.       - xpack.security.http.ssl.key=certs/es03/es03.key
    168.       - xpack.security.http.ssl.certificate=certs/es03/es03.crt
    169.       - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
    170.       - xpack.security.http.ssl.verification_mode=certificate
    171.       - xpack.security.transport.ssl.enabled=true
    172.       - xpack.security.transport.ssl.key=certs/es03/es03.key
    173.       - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
    174.       - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
    175.       - xpack.security.transport.ssl.verification_mode=certificate
    176.       - xpack.license.self_generated.type=${LICENSE}
    177.     mem_limit: ${MEM_LIMIT}
    178.     ulimits:
    179.       memlock:
    180.         soft: -1
    181.         hard: -1
    182.     healthcheck:
    183.       test:
    184.         [
    185.           "CMD-SHELL",
    186.           "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
    187.         ]
    188.       interval: 10s
    189.       timeout: 10s
    190.       retries: 120
    191.  
    192.   kibana:
    193.     depends_on:
    194.       es01:
    195.         condition: service_healthy
    196.       es02:
    197.         condition: service_healthy
    198.       es03:
    199.         condition: service_healthy
    200.     image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    201.     volumes:
    202.       # - ./kibana.yml:/usr/share/kibana/config/kibana.yml
    203.       - certs:/usr/share/kibana/config/certs
    204.       - kibanadata:/usr/share/kibana/data
    205.     ports:
    206.       - ${KIBANA_PORT}:5601
    207.     environment:
    208.       - SERVERNAME=kibana
    209.       - ELASTICSEARCH_HOSTS=https://es01:9200
    210.       - ELASTICSEARCH_USERNAME=kibana_system
    211.       - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
    212.       - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    213.     mem_limit: ${MEM_LIMIT}
    214.     healthcheck:
    215.       test:
    216.         [
    217.           "CMD-SHELL",
    218.           "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
    219.         ]
    220.       interval: 10s
    221.       timeout: 10s
    222.       retries: 120
    223.  
    224. volumes:
    225.   certs:
    226.     driver: local
    227.   esdata01:
    228.     driver: local
    229.   esdata02:
    230.     driver: local
    231.   esdata03:
    232.     driver: local
    233.   kibanadata:
    234.     driver: local

    启动集群

    在上面，我们已经创建好了如下的文件：

    1. $ pwd
    2. /Users/liuxg/data/elastic8
    3. $ ls -al
    4. total 40
    5. drwxr-xr-x    5 liuxg  staff   160 Oct 11 08:07 .
    6. drwxr-xr-x  166 liuxg  staff  5312 Nov 15 14:32 ..
    7. -rw-r--r--    1 liuxg  staff   728 Nov 17 07:13 .env
    8. -rw-r--r--    1 liuxg  staff  8321 Nov 17 07:30 docker-compose.yml

    我们使用如下的命令来启动 Elasticsearch 集群：

    docker-compose up

    或者：

    docker-compose up -d

    如果你想让 docker-compose 在后台运行的话。

    直到我们看到如下的画面：

     

    我们在浏览器中启动 Kibana：

     

    在上面，我们输入密码 password 即可登录：

     

    我们在 console 中打入如下的命令：

    GET _cat/nodes

    我们可以看到如下的结果：

    1. 172.21.0.5 43 100 10 0.33 0.89 0.66 cims - es03
    2. 172.21.0.3 49  96  9 0.33 0.89 0.66 hims - es01
    3. 172.21.0.4 72 100 10 0.33 0.89 0.66 imsw * es02

     

    如果你不想使用 Kibana，你也可以使用 curl 命令来查看：

    curl -k -u elastic:password https://localhost:9200/_cat/nodes
    1. $ curl -k -u elastic:password https://localhost:9200/_cat/nodes
    2. 172.21.0.5 77 100 4 0.27 0.62 0.59 cims - es03
    3. 172.21.0.3 64  97 4 0.27 0.62 0.59 hims - es01
    4. 172.21.0.4 69 100 4 0.27 0.62 0.59 imsw * es02

    或者更为确切的显示：

    GET _cat/nodes?v

    从上面的输出中，我们可以看到每个节点的 role。如果我们想了解上面的 cims 各个代表什么意义，请参阅另外一篇文章 “Elasticsearch：Node roles 介绍 - 7.9 之后版本”。字母的意思是：

    • c : cold node
    • d : data node
    • f : frozen node
    • h : hot node
    • i : ingest node
    • l : machine learning node
    • m : master eligible node
    • r : remote cluster client node
    • s : content node
    • t : transform node
    • v : voting-only node
    • w : warm node
    • -  : coordinating node only
       

    比如针对 es02 这个节点，imsw 代表的意思是：ingest node，master eligible node，content node，warm node。

    这样我们就创建了我们所需要的热，温及冷架构的 Elasticsearch 集群。我们可以在这个集群上测试我们的 ILM（索引生命周期管理）练习。

  • 相关阅读:
    Tomcat学习
    【图像处理-计算机视觉学习路线】个人记录
    【爬虫】第二部分 urllib_handler处理器
    C位操作符
    Tomcat部署war包项目请求404
    企业AI虚拟ip形象定制的应用场景
    25 Fisher判别
    ZHOJ_#20952. 最大公约数_数论
    VSCode常用快捷键记录
    Dart HttpClient 网络请示框架的使用详解
  • 原文地址：https://blog.csdn.net/UbuntuTouch/article/details/127896705