TiKV --force Forced Offline and Store Deletion

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv --force强制下线后,stroe删除

| username: jackerzhou

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version] v6.5.0
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Issue Phenomenon and Impact]
After forcing offline with tiup --force, the corresponding data directory has been deleted.
tiup ctl:v6.5.0 pd -u http://10.64.24.25:2379 store delete 2 shows success, but store 2 still cannot be deleted.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
{
“store”: {
“id”: 2,
“address”: “192.168.1.1:20160”,
“version”: “6.5.0”,
“peer_address”: “192.168.1.1:20160”,
“status_address”: “192.168.1.1:20180”,
“git_hash”: “47b81680f75adc4b7200480cea5dbe46ae07c4b5”,
“start_timestamp”: 1679556798,
“deploy_path”: “/mnt/tidb-deploy/tikv-20160/bin”,
“last_heartbeat”: 1679536854136206516,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 2,
“region_weight”: 1,
“region_score”: 2,
“region_size”: 2,
“witness_count”: 0,
“slow_score”: 0,
“start_ts”: “2023-03-23T15:33:18+08:00”,
“last_heartbeat_ts”: “2023-03-23T10:00:54.136206516+08:00”
}
}

| username: weixiaobing | Original post link

“region_count”: 2 There are still 2 regions that have not been migrated. You can check the scheduling test or other aspects.

| username: h5n1 | Original post link

Take a look at the regions that haven’t been migrated: pd-ctl region store 2. Check if the region status is normal: 3 voters, one of which is the Leader. If there is a leader, use pd-ctl operator add remove-peer. If there is no leader or the region status is incorrect, consider unsafe recovery or recreate.
Reference:

| username: hey-hoho | Original post link

You need to remove the remaining regions first. Try using a small tool I wrote that can remove them in batches:
clear_tikv_regions.html (7.2 KB)

After removal, the store will become Tombstone status, then you can use store delete.

| username: jackerzhou | Original post link

How do you view the region JSON in the tool?

| username: hey-hoho | Original post link

After entering the pd ctl information and store id, a command will be generated. Execute it to obtain region.json.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.