Help: Tombstone Stores

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 求助: Tombstone Stores

| username: porpoiselxj

[Test Environment] TiDB
[TiDB Version] v6.1.1
[Reproduction Path] What operations were performed to cause the issue
[Encountered Issue: Issue Phenomenon and Impact]
The PD monitoring panel shows 6 Tombstone Stores in the Abnormal stores section, while other metrics are normal and the service is temporarily unaffected. Seeking the cause of the issue, its location, and solution, detailed as shown in the image below:

| username: Kongdom | Original post link

Have you performed a scale-in?

  • Operations on TiKV, TiFlash, and TiDB Binlog components:
    • tiup-cluster directly exits after taking them offline through the API without waiting for the offline process to complete
    • Execute tiup cluster display to check the status of the offline nodes and wait for their status to change to Tombstone
    • Execute the tiup cluster prune command to clean up Tombstone nodes, which will perform the following operations:
      • Stop the services of the nodes that have been taken offline
      • Clean up the related data files of the nodes that have been taken offline
      • Update the cluster topology to remove the nodes that have been taken offline
| username: porpoiselxj | Original post link

The node information displayed by tiup cluster display is all normal (all are UP). This cluster has only been scaled out and has not yet undergone any scaling in or decommissioning operations.

| username: Kongdom | Original post link

Uh… I haven’t encountered this before. Normally, those in the Tombstone state can be cleaned up. If you want to keep the interface tidy, you can try executing the tiup cluster prune command.

| username: h5n1 | Original post link

Has this cluster been scaled down before? Try using pd-ctl store remove-tombstone to clean it up.

| username: porpoiselxj | Original post link

This works, restored, thanks a lot.

| username: Raymond | Original post link

This cluster should have been scaled down before?

| username: xingzhenxiang | Original post link

This way, there is no need to install the pd-ctl tool separately.

curl -X DELETE {PDLeaderIP}:2379/pd/api/v1/stores/remove-tombstone

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.