PD cannot delete tombstone store nodes

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: pd无法删除tombstone的store节点

| username: 华健-梦诚科技

[TiDB Usage Environment] Production Environment
[TiDB Version] 7.1.2

There was an issue with 3 TiFlash nodes, manually scaled down, then forcibly deleted in PD, and they became tombstone status.

The cluster has returned to normal, but these 3 tombstone stages cannot be deleted. The error occurs when deleting:

» store remove-tombstone
Failed to remove tombstone store [500] "failed stores: 7512016, 7512019, 7512017"

The store information is as follows:

» store 7512016
  "store": {
    "id": 7512016,
    "address": "tidb01:3930",
    "labels": [
        "key": "engine",
        "value": "tiflash"
        "key": "host",
        "value": "data01"
        "key": "region",
        "value": "mctech"
        "key": "zone",
        "value": "tc-beijing"
    "version": "v7.1.2",
    "peer_address": "tidb01:20170",
    "status_address": "tidb01:20292",
    "git_hash": "1b60452040258606e96b830b040aabf54625a8f3",
    "start_timestamp": 1703267460,
    "deploy_path": "/tidb-deploy/tiflash-9000/bin/tiflash",
    "last_heartbeat": 1703264497055671655,
    "node_state": 3,
    "state_name": "Tombstone"
  "status": {
    "capacity": "0B",
    "available": "0B",
    "used_size": "0B",
    "leader_count": 0,
    "leader_weight": 1,
    "leader_score": 0,
    "leader_size": 0,
    "region_count": 0,
    "region_weight": 1,
    "region_score": 0,
    "region_size": 0,
    "start_ts": "2023-12-23T01:51:00+08:00",
    "last_heartbeat_ts": "2023-12-23T01:01:37.055671655+08:00"

The corresponding warning information can be found in the PD log, but nothing can be seen:

[2023/12/23 04:16:06.073 +08:00] [WARN] [cluster.go:2032] ["skip removing tombstone"] [store="id:7512016 address:\"tidb01:3930\" state:Tombstone labels:<key:\"engine\" value:\"tiflash\" > labels:<key:\"host\" value:\"data01\" > labels:<key:\"region\" value:\"mctech\" > labels:<key:\"zone\" value:\"tc-beijing\" > version:\"v7.1.2\" peer_address:\"tidb01:20170\" status_address:\"tidb01:20292\" git_hash:\"1b60452040258606e96b830b040aabf54625a8f3\" start_timestamp:1703267460 deploy_path:\"/tidb-deploy/tiflash-9000/bin/tiflash\" last_heartbeat:1703264497055671655 node_state:Removed "]

Is there any way to delete these 3 tombstone stores?

| username: h5n1 | Original post link

What is the specific process for manual scaling down?

| username: wangccsy | Original post link

I applied for the use of a cloud server, but I don’t really know how to use it.

| username: heiwandou | Original post link

Please share the deletion process.

| username: 小龙虾爱大龙虾 | Original post link

How did it end up like this? Please explain.

| username: 华健-梦诚科技 | Original post link

Upgraded the server’s operating system and restarted the machine. Then, three TiFlash nodes couldn’t start, continuously reporting errors and restarting in a loop. I didn’t save the error screenshots, but it was roughly that these three nodes couldn’t communicate with each other.

So, I had the operations team use pd-ctl to forcefully scale down because the normal scale-down was always in a pending state and still kept reporting errors and couldn’t start.

When scaling up, it reported an error saying that these three stores still existed:
[2023/12/23 01:53:34.636 +08:00] [FATAL] [run.rs:1267] ["failed to start node: Other(\"[components/pd_client/src/util.rs:885]: duplicated store address: id:445645339 address:\\\"tidb01:3930\\\" labels:<key:\\\"engine\\\" value:\\\"tiflash\\\" > version:\\\"v7.1.2\\\" peer_address:\\\"tidb01:20170\\\" status_address:\\\"tidb01:20292\\\" git_hash:\\\"1b60452040258606e96b830b040aabf54625a8f3\\\" start_timestamp:1703267614 deploy_path:\\\"/tidb-deploy/tiflash-9000/bin/tiflash\\\" , already registered by id:7512016 address:\\\"tidb01:3930\\\" state:Offline labels:<key:\\\"engine\\\" value:\\\"tiflash\\\" > labels:<key:\\\"host\\\" value:\\\"data01\\\" > labels:<key:\\\"region\\\" value:\\\"mctech\\\" > labels:<key:\\\"zone\\\" value:\\\"tc-beijing\\\" > version:\\\"v7.1.2\\\" peer_address:\\\"tidb01:20170\\\" status_address:\\\"tidb01:20292\\\" git_hash:\\\"1b60452040258606e96b830b040aabf54625a8f3\\\" start_timestamp:1703267460 deploy_path:\\\"/tidb-deploy/tiflash-9000/bin/tiflash\\\" last_heartbeat:1703264497055671655 node_state:Removing \")"]

So, I used pd-ctl to delete these three nodes, then scaled up, and it worked. Now the cluster is functioning normally, but these three tombstones remain and can’t be cleared.

| username: Jellybean | Original post link

The suspected cause of this issue is due to the unreasonable forced deletion step by pd-ctl during the removal process.

Please confirm whether the columnar replica was set to 0 before removing the old TiFlash node? If you want to take TiFlash offline, you must set this first to stop the real-time data synchronization from TiKV to TiFlash.

| username: 华健-梦诚科技 | Original post link

The TiFlash replica was not set to 0. At that time, the TiFlash on these three nodes could not start and was always in a disconnected state, so it should not have been synchronizing with TiKV.

| username: Jellybean | Original post link

This is one of the issues with scaling down that couldn’t be resolved before.

If you want to scale down all TiFlash nodes, even if there is no data flow at the underlying level, you need to set the columnar replica to 0 first, otherwise, it cannot be scaled down.

Moreover, even if the node cannot start, the scaling down operation should be executed through tiup, and should not be directly operated using pd-ctl.

| username: 华健-梦诚科技 | Original post link

I see, I’ve learned something new, thank you.

The reason for not setting it to 0 is because TiFlash synchronization is particularly slow. If set to 0, and then adjusted back after tinkering, there might not be enough time. This operation is done in the production environment at night, and it must be restored to normal usability by morning.

So how should we clean up the tombstones now? Can any experts provide some guidance?

| username: tidb菜鸟一只 | Original post link

Try using --force with tiup to forcibly clean it up.

| username: 连连看db | Original post link

Execute the tiup cluster prune command to clean up Tombstone nodes.

| username: Jellybean | Original post link

What is the current status of the cluster as viewed through tiup display? Check if it has entered the Tombstone state. If it has, you can try using tiup cluster prune to clean it up.