Can a mistakenly deleted TiKV be re-added to the cluster?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv被误删除,还能加回集群吗?

| username: Martins

[TiDB Usage Environment] Production Environment
[TiDB Version] v7.1.1
[Reproduction Path] The server’s memory usage is too high, the kv process is killed by the system and cannot be started, indicating data loss. After mistakenly operating unsafe remove-failed-stores, it prompts store is tombstone when starting.
[Encountered Problem: Problem Phenomenon and Impact] How to revoke the unsafe remove-failed-stores command
[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]

| username: 像风一样的男子 | Original post link

Since this node has been deleted, just expand it back in place, and the data will automatically balance itself.

| username: tidb菜鸟一只 | Original post link

If it’s already in the tombstone state, you can only scale down and then scale up again. If it’s just offline and not completely down, you can manually bring it online… curl -X POST http://127.0.0.1:2379/pd/api/v1/store/49317/state?state=Up

| username: Martins | Original post link

Okay, thank you!

| username: 小龙虾爱大龙虾 | Original post link

If you only set up one TiKV node, then it’s fine :joy_cat:, just scale it back up and it will be okay.

| username: dba远航 | Original post link

First, scale down and then scale up.

| username: andone | Original post link

The TiKV node can be expanded back.

| username: zxgaa | Original post link

Let’s expand one more.

| username: Jellybean | Original post link

The status of the TiKV node is tombstone, which means it has been “marked with a tombstone” and cannot “come back to life.” It also indicates that the data has been migrated to other nodes.

Therefore, you can directly execute tiup prune to remove the related information, and then use the same machine, directory, and port to expand a new node, completing the “regeneration.”

After that, the cluster will automatically schedule the data from other nodes to maintain a balanced state.

| username: TIDB-Learner | Original post link

I have a question, can any expert provide some guidance?
OP, manually shutting down the TiKV instance process, I think, is equivalent to an unexpected program exit. By using the start command to start this instance, it should be able to start, right? Why is it in the tombstone state?

| username: 普罗米修斯 | Original post link

Because the command unsafe remove-failed-stores was executed, all regions on this node have been migrated out, so it is in tombstone mode.

| username: TIDB-Learner | Original post link

This column is quite relevant to the topic starter’s question.