• 小型架构实验模拟


    一  实验需求

    二    实验环境

    22 机器: 做nginx 反向代理  做静态资源服务器

    装 nginx  keepalived filebeat

    44机器: 做22 机器的备胎

    装nginx  keepalived 

    99机器:做mysql的主

    装mysqld   装node 装filebeat

    77机器:做mysql从    做tomcat动态资源服务器

    装mysqld node tomcat

    88机器: 做mysql从  做tomcat 动态资源服务器

    装mysqld node1 tomcat

    66 机器: 做elk 节点服务器   且做mha 的manager 节点

    装logstash es  es-head  kibana  manager

    三   实验步骤

    1,先做nginx的反向代理  和不同域名指向

    1.1(22,44)nginx 配置文件如下:
    1. http {
    2. log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    3. '$status $body_bytes_sent "$http_referer" '
    4. '"$http_user_agent" "$http_x_forwarded_for"';
    5. access_log /var/log/nginx/access.log main;
    6. sendfile on;
    7. tcp_nopush on;
    8. tcp_nodelay on;
    9. keepalive_timeout 65;
    10. types_hash_max_size 4096;
    11. upstream tomcat {
    12. server 192.168.217.77:8080;
    13. server 192.168.217.88:8080;
    14. }
    15. include /etc/nginx/mime.types;
    16. default_type application/octet-stream;
    17. # Load modular configuration files from the /etc/nginx/conf.d directory.
    18. # See http://nginx.org/en/docs/ngx_core_module.html#include
    19. # for more information.
    20. include /etc/nginx/conf.d/*.conf;
    21. server {
    22. listen 80;
    23. listen [::]:80;
    24. server_name _;
    25. root /usr/share/nginx/html;
    26. location ~*\.jsp$ {
    27. proxy_pass http://tomcat;
    28. }
    29. location ~* \.(jpg|jpeg|png|gif|bmp|html)$ {
    30. root /usr/share/nginx/html/;
    31. }
    32. }
    33. server {
    34. listen 80;
    35. server_name www.benet.com;
    36. root /data/benet;
    37. location / {
    38. proxy_pass http://192.168.217.77:8080;
    39. }
    40. }
    41. server {
    42. listen 80;
    43. server_name www.kgc.com;
    44. root /data/kgc;
    45. location / {
    46. proxy_pass http://192.168.217.88:8080;
    47. }
    48. }
    49. # Load configuration files for the default server block.
    50. include /etc/nginx/default.d/*.conf;
    51. server {
    52. listen 80;
    53. error_page 404 /404.html;
    54. location = /404.html {
    55. }
    56. error_page 500 502 503 504 /50x.html;
    57. location = /50x.html {
    58. }
    59. }
    1.2 解释

    1.3 实验效果

    访问nginx 以.jsp 结尾    会以轮询方式访问  后面的tomcat

     访问 nginx  以.png 结尾   会在本地找

    访问 benet 域名   会指向tomcat1  (记得本机做域名映射)

    访问kgc 域名   会指向 tomcat 2  

    以上答案错误!   正确答案:

    客户只知道域名   根据域名做对应的动静分离

    2, 做nginx 的高可用 

    2.1 keepalive 配置文件如下 (这是主)

    备改 router_id LVS_02     state BACKUP  priority 80

    1. ! Configuration File for keepalived
    2. global_defs {
    3. notification_email {
    4. acassen@firewall.loc
    5. failover@firewall.loc
    6. sysadmin@firewall.loc
    7. }
    8. notification_email_from Alexandre.Cassen@firewall.loc
    9. smtp_server 127.0.0.1
    10. smtp_connect_timeout 30
    11. router_id LVS_01
    12. vrrp_skip_check_adv_addr
    13. vrrp_garp_interval 0
    14. vrrp_gna_interval 0
    15. }
    16. vrrp_script check_down {
    17. script "/etc/keepalived/ng.sh"
    18. interval 1
    19. weight -30
    20. fall 3
    21. rise 2
    22. timeout 2
    23. }
    24. vrrp_instance VI_1 {
    25. state MASTER
    26. interface ens33
    27. virtual_router_id 51
    28. priority 100
    29. advert_int 1
    30. authentication {
    31. auth_type PASS
    32. auth_pass 1111
    33. }
    34. virtual_ipaddress {
    35. 192.168.217.188
    36. }
    37. track_script {
    38. check_down
    39. }
    40. }
    2.2  (22,44机器)写检测nginx 是否活着的脚本(注意路径)

    2.3  实验效果

    虚拟ip 188 会飘在 22 机器上(主)

    当手动制造 22 机器故障 systemctl stop nginx    虚拟ip 会飘到44机器上

    证明:若主出现故障   从会顶上来

    3, 做mysql 主从复制与读写分离 mha做高可用

    前篇已详细概述,本篇暂不赘述 

    4,做filebeat 收集nginx 的正确错误日志

    4.1 (22机器)filebeat 的配置文件
    1. #=========================== Filebeat inputs =============================
    2. filebeat.inputs:
    3. # Each - is an input. Most options can be set at the input level, so
    4. # you can use different inputs for various configurations.
    5. # Below are the input specific configurations.
    6. - type: log
    7. enabled: true
    8. paths:
    9. - /var/log/nginx/access.log
    10. tags: ["access"]
    11. - type: log
    12. enabled: true
    13. paths:
    14. - /var/log/nginx/error.log
    15. tags: ["error"]
    1. #----------------------------- Logstash output --------------------------------
    2. output.logstash:
    3. # The Logstash hosts
    4. hosts: ["192.168.217.66:5044"]
    5. # Optional SSL. By default is off.
    6. # List of root certificates for HTTPS server verifications
    7. #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    8. # Certificate for SSL client authentication
    9. #ssl.certificate: "/etc/pki/client/cert.pem"
    10. # Client Certificate Key
    11. #ssl.key: "/etc/pki/client/cert.key"
    4.2  (66机器)logstash 配置文件
    1. input {
    2. beats {
    3. port => "5044"
    4. }
    5. }
    6. output {
    7. if "access" in [tags] {
    8. elasticsearch {
    9. hosts => ["192.168.217.66:9200"]
    10. index => "nginx-access-%{+YYYY.MM.dd}"
    11. }
    12. }
    13. if "error" in [tags] {
    14. elasticsearch {
    15. hosts => ["192.168.217.66:9200"]
    16. index => "nginx-error-%{+YYYY.MM.dd}"
    17. }
    18. }
    19. stdout {
    20. codec => rubydebug
    21. }
    22. }
    4.3 (66机器)elasticsearch主配置文件
    1. [root@node ~]#grep -v "^#" /etc/elasticsearch/elasticsearch.yml
    2. cluster.name: my-elk-cluster
    3. node.name: node
    4. path.data: /data/elk_data
    5. path.logs: /var/log/elasticsearch/
    6. bootstrap.memory_lock: false
    7. network.host: 0.0.0.0
    8. http.port: 9200
    9. discovery.zen.ping.unicast.hosts: ["node"]
    10. http.cors.enabled: true #开启跨域访问支持,默认为 false
    11. http.cors.allow-origin: "*" #指定跨域访问允许的域名地址为所

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    4.4  (66机器)kibana 配置文件
    1. [root@node ~]#grep -v "^#" /etc/kibana/kibana.yml
    2. server.port: 5601
    3. server.host: "0.0.0.0"
    4. elasticsearch.url: "http://192.168.217.66:9200"
    5. kibana.index: ".kibana"

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    4.5 实验效果

    5,  filebeat 收集mysql 的错误日志

    5.1  (99机器)mysql 配置文件添加 错误日志路径
    1. [client]
    2. port = 3306
    3. #default-character-set=utf8
    4. socket = /usr/local/mysql/mysql.sock
    5. [mysql]
    6. port = 3306
    7. default-character-set=utf8
    8. socket = /usr/local/mysql/mysql.sock
    9. [mysqld]
    10. #log-error = /var/log/mysqld/error.log
    11. user = mysql
    12. basedir = /usr/local/mysql
    13. datadir = /usr/local/mysql/data
    14. port = 3306
    15. log_error=/usr/local/mysql/error.log
    16. character_set_server=utf8
    17. pid-file = /usr/local/mysql/mysqld.pid
    18. socket = /usr/local/mysql/mysql.sock
    19. server-id = 1
    20. log_bin = master-bin
    21. log-slave-updates = true
    22. sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,PIPES_AS_CONCAT,ANSI_QUOTES
    23. ~
    24. ~

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    5.2  (99机器) filebeat 的配置文件
    1. #=========================== Filebeat inputs =============================
    2. filebeat.inputs:
    3. # Each - is an input. Most options can be set at the input level, so
    4. # you can use different inputs for various configurations.
    5. # Below are the input specific configurations.
    6. - type: log
    7. # Change to true to enable this input configuration.
    8. enabled: true
    9. # Paths that should be crawled and fetched. Glob based paths.
    10. paths:
    11. - /usr/local/mysql/error.log
    12. tags: ["error"]
    13. fields: #可以使用 fields 配置选项设置一些参数字段添加到 output 中
    14. service_name: filebeat
    15. log_type: log
    16. service_id: 192.168.217.99

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    1. #----------------------------- Logstash output --------------------------------
    2. output.logstash:
    3. # The Logstash hosts
    4. hosts: ["192.168.217.66:5044"]

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    5.3  (66 机器)写logstash 配置文件
    1. input {
    2. beats {
    3. port => "5044"
    4. }
    5. }
    6. output {
    7. elasticsearch {
    8. hosts => ["192.168.217.66:9200"]
    9. index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
    10. }
    11. stdout {
    12. codec => rubydebug
    13. }
    14. }

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    5.4  实验结果

    四   报错总结

    1,logstash 配置文件占用报错

    1. [root@node logstash]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/mysql.conf
    2. ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
    3. WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
    4. Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
    5. 07:40:53.541 [LogStash::Runner] FATAL logstash.runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

    wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

    解决方法:

    找到主进程  kill-9 93968

    2,filebeat 起不来, 配置文件报错

    软件启动不来   看 /var/log/messages

    filebeat 的配置文件确保所有键值对的值正确缩进,遵循YAML的缩进规则(通常使用空格,且同一层级的元素应保持相同数量的空格)。说人话就是注意空格

    3,manager 节点上测试 mysql 主从连接情况 MySQL Replication Health is not OK

    masterha_check_repl -conf=/etc/masterha/app1.cnf
     

    解决方法:

    4, mysql 配置文件 加不了错误日志路径

    解决方法:编译安装的mysql 环境变量如下

    所以日志也要放在该路径下

    echo 'PATH=/usr/local/mysql/bin:/usr/local/mysql/lib:$PATH' >> /etc/profile
    echo 'export PATH' >> /etc/profile
    source /etc/profile

    5, 做mha 时  manager 节点报错

     manager 节点也要 先装node

    6,做mysql 的elfk 时一直出不来

    注意数据流向   ip 是否写错

    7  mha 故障恢复

    7.1 故障恢复步骤
    1. 故障修复步骤:
    2. 1.修复mysql
    3. systemctl restart mysqld
    4. 2.修复主从
    5. #在现主库服务器 Mysql2 查看二进制文件和同步点
    6. show master status;
    7. #在原主库服务器 mysql1 执行同步操作
    8. change master to master_host='192.168.217.77',master_user='myslave',master_password='123456',master_log_file='master-bin.000002',master_log_pos=1745;
    9. start slave;
    10. 3.在 manager 节点上修改配置文件app1.cnf(再把这个记录添加进去,因为它检测掉失效时候会自动消失)
    11. vi /etc/masterha/app1.cnf
    12. ......
    13. secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.10.14 -s 192.168.10.15
    14. ......
    15. [server1]
    16. hostname=192.168.217.77
    17. port=3306
    18. [server2]
    19. candidate_master=1
    20. check_repl_delay=0
    21. hostname=192.168.217.88
    22. port=3306
    23. [server3]
    24. hostname=192.168.217.99
    25. port=3306
    26. 4.在 manager 节点上启动 MHA
    27. nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

    7.2  报错

    报错1   app1 is stopped(2:NOT_RUNNING).

    1. [root@node ~]#masterha_check_status --conf=/etc/masterha/app1.cnf
    2. 这是查看 mha 主状态
    3. app1 is stopped(2:NOT_RUNNING).


    报错2   MySQL Replication Health is NOT OK!

    1. [root@node ~]#masterha_check_repl -conf=/etc/masterha/app1.cnf
    2. 这是检测mha 主从复制情况
    3. Mon Apr 22 01:13:23 2024 - [warning] Global configuration file /etc/masterha_default.cnf not found.
    4. Mon Apr 22 01:13:23 2024 - [info] Reading application default configuration from /etc/masterha/app1.
    5. Mon Apr 22 01:13:23 2024 - [info] Reading server configuration from /etc/masterha/app1.cnf..
    6. Mon Apr 22 01:13:23 2024 - [info] MHA::MasterMonitor version 0.57.
    7. Mon Apr 22 01:13:24 2024 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configurations for details. Master configurations are as below:
    8. Master 192.168.217.77(192.168.217.77:3306), replicating from 192.168.217.99(192.168.217.99:3306)
    9. Master 192.168.217.99(192.168.217.99:3306), replicating from 192.168.217.77(192.168.217.77:3306)
    10. Mon Apr 22 01:13:24 2024 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happene
    11. Mon Apr 22 01:13:24 2024 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happene
    12. Mon Apr 22 01:13:24 2024 - [info] Got exit code 1 (Not master dead).
    13. MySQL Replication Health is NOT OK!
    14. [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_m

    7.3  解决办法

    先看报错 日志:

    [root@node ~]#masterha_check_repl -conf=/etc/masterha/app1.cnf Mon Apr 22 01:13:23

    2024 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Mon Apr 22 01:13:23 2024 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Mon Apr 22 01:13:23 2024 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Mon Apr 22 01:13:23 2024 - [info] MHA::MasterMonitor version 0.57. Mon Apr 22 01:13:24 2024 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configuration is detected, but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. Master configurations are as below: Master 192.168.217.77(192.168.217.77:3306), replicating from 192.168.217.99(192.168.217.99:3306) Master 192.168.217.99(192.168.217.99:3306), replicating from 192.168.217.77(192.168.217.77:3306)

    Mon Apr 22 01:13:24 2024 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 329. Mon Apr 22 01:13:24 2024 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers. Mon Apr 22 01:13:24 2024 - [info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK! [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 [root@node ~]#masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).

    您提供的最新输出显示,在运行masterha_check_replmasterha_check_status命令时,MySQL复制健康状况依然不正常,并且MHA(Master High Availability Manager)应用app1的状态为停止(stopped(2:NOT_RUNNING))。以下是针对输出中关键信息的中文解释:

    1. 警告Global configuration file /etc/masterha_default.cnf not found. Skipping. 表示系统未能找到全局配置文件 /etc/masterha_default.cnf。虽然跳过了该文件的读取,但建议检查文件是否存在,以及路径是否正确,确保所有必要的配置项已正确设置。

    2. 信息Reading application default configuration from /etc/masterha/app1.cnf.. 和 Reading server configuration from /etc/masterha/app1.cnf.. 表明正在从 /etc/masterha/app1.cnf 文件中读取应用默认配置和服务器配置

    3. 信息MHA::MasterMonitor version 0.57. 显示当前使用的 MHA Master Monitor 版本为 0.57。

    4. 错误Multi-master configuration is detected, but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. 提示检测到存在多主(multi-master)配置,且有两个或更多主服务器处于可写状态(未设置为只读)或已经失效。这是一个严重问题,因为在一个典型的主从复制环境中,应该只有一个主服务器是可写的,其他服务器作为从服务器接收主服务器的数据复制。建议仔细检查以下两台主机的配置:

      • Master 192.168.217.77(192.168.217.77:3306), replicating from 192.168.217.99(192.168.217.99:3306)
      • Master 192.168.217.99(192.168.217.99:3306), replicating from 192.168.217.77(192.168.217.77:3306)

      确保每个服务器的角色(主/从)正确设定,并且仅有一个主服务器处于可写状态。如果需要,可以检查并调整 read_only 系统变量,确保从服务器设置为只读。

    5. 错误Error happened on checking configurations. at /usr/local/share/perl5/MHA/ServerManager.pm line 329. 和 Error happened on monitoring servers. 表示在检查配置和监控服务器时发生了错误。这些错误可能与上述多主配置问题直接相关。

    6. 信息Got exit code 1 (Not master dead). 说明命令执行以退出码 1 结束,表示主服务器并未失效,但鉴于前面的错误信息,实际上是由于检测到了不正常的多主配置导致的。

    7. MHA状态app1 is stopped(2:NOT_RUNNING). 表明MHA应用app1当前处于停止状态,无法正常监控和管理MySQL复制环境的高可用性。

    综上所述,您需要首先解决多主配置问题,确保只有单一主服务器处于可写状态,其余服务器作为从服务器正确地从主服务器接收复制。修复配置问题后,重启MHA应用app1,然后再次运行 masterha_check_repl 和 masterha_check_status 命令,以确认复制健康状况已恢复正常,并且MHA应用能够成功运行并监控您的MySQL复制环境。如果问题仍未解决,请进一步检查MHA配置文件和日志,以获取更详细的故障信息。

    在原来的  主 上   做 只读 

    成功!

    8    mha 故障恢复 从节点顶不上来

    每个mysql 上都有加这个   因为每个mysql 都有可能成为从   和  主

    1. [client]
    2. port = 3306
    3. #default-character-set=utf8
    4. socket = /usr/local/mysql/mysql.sock
    5. [mysql]
    6. port = 3306
    7. default-character-set=utf8
    8. socket = /usr/local/mysql/mysql.sock
    9. [mysqld]
    10. user = mysql
    11. basedir = /usr/local/mysql
    12. datadir = /usr/local/mysql/data
    13. port = 3306
    14. character_set_server=utf8
    15. pid-file = /usr/local/mysql/mysqld.pid
    16. socket = /usr/local/mysql/mysql.sock
    17. server-id = 2
    18. log_bin = master-bin
    19. relay-log = relay-log-bin
    20. relay-log-index = slave-relay-bin.index
    21. sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,PIPES_AS_CONCAT,ANSI_QUOTES

  • 相关阅读:
    如何将 Apache Airflow 用于机器学习工作流
    Istio1.12:安装和快速入门
    一文吃透接口调用神器RestTemplate
    vue 城市选择器的使用 element-china-area-data
    INC公司和LLC公司的区别
    Visual Studio Code如何写C语言
    Linux 学习笔记(2)—— 关于文件和目录
    android源码设计模式学习笔记(一)
    【IOS-初学】利用分段选择器和滑动开关(条)等等实现简单的图片浏览器-透明度和图片切换功能
    信息系统项目管理师考试重点汇总,看完这篇再拿十分!
  • 原文地址:https://blog.csdn.net/wyq2070857323/article/details/138035954