How to Remove Offline Status TiKV

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何删除offline状态tikv

| username: usretyn

[TiDB Usage Environment] Production Environment
[TiDB Version] v5.3.1 3 replicas
[Reproduction Path] What operations were performed when the issue occurred
Accidentally forced to scale in one KV node using the command: tiup scale-in tidb -N 172.16.61.21:20160 --force -y
Then scaled out one KV node again, and the current cluster status is normal.
[Encountered Issue: Problem Phenomenon and Impact]
Using tiup to check cluster information shows no scaled-in nodes. Using pd-ctl stores to get scaled-in node information shows the status as offline, with a region count of 120.

The data directory no longer exists after scaling in. What method can be used to delete this node?
{
“store”: {
“id”: 4,
“address”: “172.16.61.21:20160”,
“state”: 1,
“version”: “5.3.1”,
“status_address”: “172.16.61.21:20180”,
“git_hash”: “ecc2549ed63fc21f1f6e11f1b85f19c2465234a5”,
“start_timestamp”: 1690773632,
“deploy_path”: “/opt/tidb-deploy/tikv-20160/bin”,
“last_heartbeat”: 1690773644351448028,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 120,
“region_weight”: 1,
“region_score”: 1048,
“region_size”: 1048,
“slow_score”: 0,
“start_ts”: “2023-07-31T11:20:32+08:00”,
“last_heartbeat_ts”: “2023-07-31T11:20:44.351448028+08:00”,
“uptime”: “12.351448028s”
}
}

2
| username: redgame | Original post link

Execute remove <store_id> in the PD-CTL terminal.

3
| username: usretyn | Original post link

In pd v5.3.1, there is no command pd-ctl store delete <store_id>, it can only change TiKV to offline status.
Usage:
pd-ctl [command]

Available Commands:
cluster show the cluster information
completion Output shell completion code for the specified shell (bash)
config tune pd configs
exit exit pdctl
health show all node’s health information of the pd cluster
help Help about any command
hot show the hotspot status of the cluster
label show the labels
log set log level
member show the pd member status
operator operator commands
ping show the total time spend ping the pd
plugin plugin commands
region show the region status
scheduler scheduler commands
service-gc-safepoint show all service gc safepoint
store manipulate or query stores
tso parse TSO to the system and logic time
unsafe Unsafe operations

4
| username: tidb菜鸟一只 | Original post link

unsafe remove-failed-stores 101

5
| username: TiDBer_iCdTOZ1r | Original post link

Use “remove”.

6
| username: hey-hoho | Original post link

Here’s a small tool for you. First, clear all the regions on the store, then execute store remove-tombstone.
clear_tikv_regions.html (7.7 KB)

7
| username: h5n1 | Original post link
8
| username: usretyn | Original post link

Problem solved.

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.