Unable to delete regions on offline TiKV nodes

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv offline 节点上的region无法删除

| username: Hacker_ynbNppAC

[TiDB Usage Environment] Production Environment
[TiDB Version] 4.0
[Reproduction Path] With 5 TiKV nodes, after mistakenly deleting the directories of 2 nodes, one node had a problematic status, so it was removed and became offline. I want it to become tombstone status and then delete it.
[Encountered Problem: Symptoms and Impact]
[Resource Configuration] 5 TiKV nodes, 3 PD nodes, 2 TiDB nodes
TiKV 2T/node

Using pd-ctl region can find regions, and there are four regions on this store.
However, the command operator add remove-peer 1 2 directly reports not found.
Found that the values of the four regions are null.

2
| username: Hacker_ynbNppAC | Original post link

Currently, there is no way to delete these four regions, all of them are not found.

3
| username: WalterWj | Original post link

There is an unsafe recover operation on asktug, go ahead and do it. It seems that multiple replica data has been lost.

4
| username: Hacker_ynbNppAC | Original post link

Now there’s a problem: after I restarted all the TiKV nodes, all the nodes became offline.

5
| username: DBRE | Original post link

In what scenarios would data be lost? :joy:

6
| username: Hacker_ynbNppAC | Original post link

curl -X POST http://xxx:2379/pd/api/v1/store/1/state?state=Stombstone then delete this node, delete the data directory and rejoin
Although these regions still cannot be deleted, this node was originally a faulty node, and only these four regions have problems. So it can be operated in this way.

7
| username: Minorli-PingCAP | Original post link

Encountered multiple replicas, need to perform unsafe recovery, data may be lost.

8
| username: h5n1 | Original post link

Try recreating the problematic regions.

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.