Tuesday, January 5, 2021

Group Replication will not rejoin group due to connection issue

 

Summary

Group replication left as a member of the group and will not rejoin.

Track down reason why it will not rejoin group, example here is it will not reconnect due to a credential issue.

 

1.       Check what error is connection issues is happening trying to rejoin the group in the local database

 

select * from performance_schema.replication_connection_status ;

 

Example:

| group_replication_recovery |                                      |                                      |      NULL | OFF           |                         0 | 0000-00-00 00:00:00.000000 |                                                                                                                                                                                                                      |             13117 | Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. | 2021-01-05 08:38:15.014830 |                         | 0000-00-00 00:00:00.000000                        | 0000-00-00 00:00:00.000000                         | 0000-00-00 00:00:00.000000                    | 0000-00-00 00:00:00.000000                  |                      | 0000-00-00 00:00:00.000000                     | 0000-00-00 00:00:00.000000                      | 0000-00-00 00:00:00.000000                 |

** In this case it is stating that empty username is being used attempting to connect in.

 

2.       check the channel definition

 

select channel_name, host, port, user, network_interface, auto_position from performance_schema.replication_connection_configuration ;

 

Example:

+----------------------------+--------------------------+------+----------+-------------------+---------------+

| channel_name               | host                     | port | user     | network_interface | auto_position |

+----------------------------+--------------------------+------+----------+-------------------+---------------+

|                            | aglqapmyp01              | 3306 | rpl_user |                   | 0             |

| group_replication_applier  | <NULL>                   |    0 |          |                   | 1             |

| group_replication_recovery | aglqapmyp05.xxxxxxxx.xxx | 3306 |          |                   | 1             |

+----------------------------+--------------------------+------+----------+-------------------+---------------+

3 rows in set (0.00 sec)

 

** Note in this case the user column is missing

 

 

3.       Since it appears that the credentials are missing for the channel we can reset them and give a username and password for the channel.

 

CHANGE MASTER TO MASTER_USER='rpl_user', MASTER_PASSWORD='xxxxxxx' FOR CHANNEL 'group_replication_recovery';

 

 

4.       Recheck that our change for the credential took affect

 

select channel_name, host, port, user, network_interface, auto_position from performance_schema.replication_connection_configuration ;

 

Example:

+----------------------------+--------------------------+------+----------+-------------------+---------------+

| channel_name               | host                     | port | user     | network_interface | auto_position |

+----------------------------+--------------------------+------+----------+-------------------+---------------+

|                            | aglqapmyp01              | 3306 | rpl_user |                   | 0             |

| group_replication_applier  | <NULL>                   |    0 |          |                   | 1             |

| group_replication_recovery | aglqapmyp05.xxxxxxxx.xxx | 3306 | rpl_user |                   | 1             |

+----------------------------+--------------------------+------+----------+-------------------+---------------+

3 rows in set (0.00 sec)

 

 

 

 

5.       Restart database to restart group replication, a good way to do that is restart the database with the mysql group replication start on boot on, this way we are completely clean.

 

Example:

/etc/rc.d/init.d/mysql.server stop

/etc/rc.d/init.d/mysql.server start

 

 

6.       Check group replication status

 

SELECT MEMBER_HOST,MEMBER_STATE,MEMBER_ROLE,MEMBER_VERSION FROM performance_schema.replication_group_members order by MEMBER_HOST;

 

Example:

+--------------------------+--------------+-------------+----------------+

| MEMBER_HOST              | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |

+--------------------------+--------------+-------------+----------------+

| aglqapmyp04.xxxxxxxx.xxx | ONLINE       | SECONDARY   | 8.0.22         |

| aglqapmyp05.xxxxxxxx.xxx | ONLINE       | PRIMARY     | 8.0.22         |

| aglqapmyp06.xxxxxxxx.xxx | ONLINE       | SECONDARY   | 8.0.22         |

+--------------------------+--------------+-------------+----------------+

No comments: