案例说明：
在一次KingbaseES V8R3集群切换分析中，运维人员执行了pcp_node_refresh，导致集群发生了failover的切换。此文档对pcp_node_refresh工具做了应用说明及测试，不建议在生产环境使用此工具。
适用版本： KingbaseES V8R3
一、pcp_node_refresh应用说明
- 1. pcp_node_refresh 必须要指定节点ID(就是show pool_nodes中查出来的id)，刷新这个节点的状态。动作是将数据库踢出集群，然后再加入进来。
- 2. pcp_node_refresh 指向主库，将主库踢出集群，这个时候会认为主库故障触发会切换 ----------- 集群failover切换。
- 3. pcp_node_refresh 指向备库，将备库踢出集群再加入，集群无影响。
二、测试案例
1、集群节点状态
- TEST=# show pool_nodes;
- node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
- ---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
- 0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | true | 0
- 1 | 192.168.1.102 | 54321 | down | 0.500000 | standby | 0 | false | 0
- (2 rows)
-
- ---如上所示，备库节点状态是down，执行pcp_node_refresh刷新节点。
2、执行pcp_node_refresh刷新备库节点
1)执行pcp_node_refresh
- # 执行pcp_node_refresh
- [kingbase@node101 bin]$ ./pcp_node_refresh -U kingbase -h 192.168.1.101 1
- Password:
- pcp_node_refresh -- Command Successful
-
- # 查看节点状态
- [kingbase@node101 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
- ksql (V008R003C002B0290)
- Type "help" for help.
-
- TEST=# show pool_nodes;
- node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
- ---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
- 0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | false | 0
- 1 | 192.168.1.102 | 54321 | up | 0.500000 | standby | 0 | true | 0
- (2 rows)
-
-
- TEST=# select * from sys_stat_replication;
- PID | USESYSID | USENAME | APPLICATION_NAME | CLIENT_ADDR | CLIENT_HOSTNAME | CLIENT_PORT | BACKEND_START | B
- ACKEND_XMIN | STATE | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REPLAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
- -------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--
- ------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
- 28587 | 10 | SYSTEM | node101 | 192.168.1.102 | | 47629 | 2022-09-05 13:42:43.870560+08 |
- | streaming | 1/E92607D0 | 1/E92607D0 | 1/E92607D0 | 1/E9260780 | 0 | async
- (1 row)
-
- ---如上所示，备库节点状态恢复正常。
2)查看日志信息
如下cluster.log所示：执行pcp_node_refresh后，对备库节点作为故障节点进行了recovery的恢复。
如下recovery.log所示，执行pcp_node_refresh后，对备库节点调用sys_rewind恢复备库节点。
3、执行pcp_node_refresh刷新主库节点
1)执行pcp_node_refresh
- [kingbase@node101 bin]$ ./pcp_node_refresh -U kingbase -h 192.168.1.101 0
- Password:
- pcp_node_refresh -- Command Successful
-
-
- TEST=# show pool_nodes;
- node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
- ---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
- 0 | 192.168.1.101 | 54321 | up | 0.500000 | standby | 0 | true | 0
- 1 | 192.168.1.102 | 54321 | up | 0.500000 | primary | 0 | false | 0
- (2 rows)
-
- ---如上所示，主备节点发生了切换。
2)查看日志信息
如下cluster.log所示：执行pcp_node_refresh后，对主库节点作为故障调用failover_stream.sh执行了failover切换。
如下failover.log所示：执行pcp_node_refresh后，failover切换过程。
- -----------------2022-09-05 13:45:42 failover beging---------------------------------------
- ----failover-stats is %H = hostname of the new master node [192.168.1.102], %P = old primary node id [1], %d = node id[0], %h = host name [192.168.1.101], %O = old primary host[192.168.1.102] %m = new master node id [1], %M = old master node id [1], %D = database cluster path [/home/kingbase/cluster/R3HA/db/data].
- ----ping trust ip
- ping trust ip 192.168.1.1 success
- ----determine whether the faulty db is master or standby
- standby down, master still 192.168.1.102
- The sys_stat_replication view result is : []
- 2022-09-05 13:45:44 sync to async
- ALTER SYSTEM
- SYS_RELOAD_CONF
- -----------------
- t
- (1 row)
-
- -----------------2022-09-05 13:45:44 failover end---------------------------------------
三、总结
pcp_node_refresh不建议在生产环境使用，对于备库节点在数据库服务和流复制状态都正常的情况下，如果用show pool_nodes显示的是‘down’状态，可以使用pcp_attach_node重新注册备库节点，使节点状态恢复正常。