[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
There are 3 PD nodes, pd1, pd2, pd3. After migrating these three PD nodes to pd4, pd5, pd6 (pd1, pd2, pd3 offline), using the command:
binlogctl -pd-urls=http://192.168.133.xxx:2379 -cmd pumps
to check, the UpdateTime does not update. However, the actual downstream data is present and business is not affected. The UpdateTime only updates after restarting this pump. Does the pump not automatically detect the PD node migration?
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots / Logs / Monitoring]
According to your description, you have migrated the original PD node to a new PD node, and when you use the command binlogctl -pd-urls=http://192.168.133.xxx:2379 -cmd pumps to check the pump status, you find that UpdateTime is not updating. However, the actual downstream data is present and business is not affected. You want to know why the pump did not automatically detect the change in the PD node.
According to the official TiDB documentation [1], when you change the PD node, you need to restart the pump to make it aware of the change. This is because the pump caches the PD node information and does not automatically update it. Therefore, when you change the PD node, you need to restart the pump to reload the PD node information.
If you want to avoid restarting the pump, you can try using the command binlogctl -pd-urls=http://192.168.133.xxx:2379 -cmd refresh-workers to refresh the pump’s worker list. This will force the pump to reload the PD node information and update its worker list. However, this method may cause a performance drop in the pump because it needs to reload all the worker information.
In summary, when you change the PD node, it is recommended to restart the pump to ensure it correctly detects the change.
This cluster’s PD has been migrated for more than a year, and the pump has not been restarted. The logs have been consistently reporting “send heartbeat failed,” but it hasn’t affected data synchronization. Since the pump’s failure to send heartbeats doesn’t seem to have an impact, what are the potential consequences of this?
I tested that if the pump, whose UpdateTime doesn’t update due to not being restarted, is migrated, it will fail, and the logs will report that the drainer response cannot be found. Are there any other impacts?
The pump hasn’t restarted, and the logs keep reporting “send heartbeat failed.”
Go check the pump status with show pump status to see if there’s any issue.
The state of “show pump status” is online, and UpdateTime is not updating, but it does not actually affect data synchronization. Data can still be synchronized to the downstream through the drainer. So, what is the purpose of this heartbeat? Restarting the pump now won’t cause any other issues, right?
|
username: TiDBer_yUoxD0vR
|
username: 
|
username: 
|
username: 
|
username: