Error Occurred When Executing store delete <store_id> During Multi-Replica Damage Repair

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 多副本损坏修复时，执行store delete <store_id>报错

| username: TiDBer_QYr0vohO

[TiDB Usage Environment] Production Environment / Testing / Poc
Testing
[TiDB Version]
v7.1.0
[Reproduction Path] What operations were performed when the issue occurred
I have three TiKV nodes (single machine deployment). I deleted the disk data of 2 TiKV nodes and then tried to repair the cluster according to 【SOP 系列 18】TiUP 环境恢复 TiKV 副本 - TiDB 的问答社区. In step 7 of the link, restarting the cluster should automatically replenish the lost data directory, but the TiKV logs reported duplicate store errors. After stopping the normal TiKV node, executing store delete <store_id> resulted in an error and could not delete. Please see the screenshot below for the error message.
[Encountered Issue: Issue Phenomenon and Impact]
Executing store delete <store_id> resulted in an error and could not delete.
[Resource Configuration]

image
image1243×521 84.9 KB

[Attachments: Screenshots/Logs/Monitoring]

image
image1252×539 121 KB

image
image1246×156 18.9 KB

2
| username: Billmay表妹 | Original post link

First, try expanding by adding two nodes.

3
| username: xfworld | Original post link

Before scaling down, the cluster must be normal and healthy…

There are two ways to restore the environment:

  1. If replicas exist, expand the nodes to meet the automatic replenishment of replicas.
  2. If replicas are lost, only restore the cluster.

Regardless of the method, the cluster must be normal to perform other operations…

4
| username: h5n1 | Original post link
  1. If you want to save trouble in the testing environment, just redeploy it.
  2. If you must restore the original cluster, follow the previous suggestion to expand by 2 TiKV instances, using different ports and paths.
  3. The command store delete <store_id> will migrate the replicas on the deleted store to other nodes. The prerequisite is that you have available nodes to accept the replicas. With a 3-replica configuration, you need 3 available TiKV instances.
5
| username: TiDBer_QYr0vohO | Original post link

Okay, thank you everyone. I added 2 more TiKV nodes, and it was indeed successful.

6
| username: tidb狂热爱好者 | Original post link

It should be insufficient replicas.

7
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.