How to Clear TombstoneStores Information of Offline Nodes

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何清除已经下线节点的tombstoneStores信息

| username: CAICAI

[TiDB Usage Environment] Test
[TiDB Version] 7.5.0

TiDB is deployed using K8S. When scaling down nodes, checking the tc information shows the following content:

tombstoneStores:
  "1":
    id: "1"
    ip: tidb-tikv-0.tidb-tikv-peer.dev.svc
    lastTransitionTime: "2024-02-21T08:49:06Z"
    leaderCount: 0
    podName: tidb-tikv-0
    state: Tombstone
  "898279":
    id: "898279"
    ip: tidb-tikv-5.tidb-tikv-peer.dev.svc
    lastTransitionTime: "2024-02-21T08:53:37Z"
    leaderCount: 0
    podName: tidb-tikv-5
    state: Tombstone

Among them, tidb-tikv-5 is the node that was taken offline, and another node, tidb-tikv-0, was also taken offline due to issues. Will this information affect the cluster? Similar issues have occurred before, causing problems when upgrading the cluster version. How can this information be cleared?

Cluster as follows:

image
image1242×300 9.3 KB

2
| username: TiDBer_jYQINSnf | Original post link

There is no impact. If you must clear it,
use pd-ctl remove-tombstone.

3
| username: Fly-bird | Original post link

pd-ctl remove-tombstone

4
| username: 像风一样的男子 | Original post link

Clear tombstone status nodes
tiup cluster prune tsp-prod-tidb-cluster
Clear tombstone status nodes in Grafana
pd-ctl -u http://0.0.0.0:2379 store remove-tombstone

5
| username: dba远航 | Original post link

tiup cluster prune tsp-prod-tidb-cluster

6
| username: lemonade010 | Original post link

pd-ctl remove-tombstone

7
| username: TiDBer_jYQINSnf | Original post link

For k8s deployment, the information in tc is obtained by the operator from the pd’s API. You can delete it using pd-ctl remove-tombstone.
Several people above mentioned tiup, and I thought I said something wrong :crazy_face:

8
| username: kelvin | Original post link

pd-ctl remove-tombstone

9
| username: wangccsy | Original post link

Restart the service

10
| username: 随便改个用户名 | Original post link

What impact will it have if tombstones are not cleaned up?

11
| username: TiDBer_jYQINSnf | Original post link

It doesn’t have much impact, it’s okay even if you don’t clean it up.

12
| username: TiDBer_小阿飞 | Original post link

store remove-tombstone