After scaling down the TiDB cluster, the Grafana dashboard still shows Tombstone Stores

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB集群缩容后grafana面板还显示有Tombstone Stores

| username: zhimadi

【TiDB Usage Environment】Production Environment
【TiDB Version】v5.4.2
【Reproduction Path】Execute scale-in of tikv node
tiup cluster scale-in xxx-cluster
tiup cluster display xxx-cluster
tiup cluster prune xxx-cluster
【Encountered Problem: Phenomenon and Impact】
The Grafana panel still shows Tombstone Stores. What is the reason? Will it have any impact, and how can it be removed?
The specific path is Overview->PD->Abnormal stores
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
image

【Attachments: Screenshots/Logs/Monitoring】

| username: xfworld | Original post link

Refer to the AI processing method:

After scaling down the TiDB cluster, if there are remnants in Prometheus, you can handle it through the following steps:

  1. Log in to the server where Prometheus is located and find the Prometheus data directory, usually under the data directory.
  2. Locate the Prometheus data directory corresponding to the scaled-down TiDB instance, typically named after the instance’s IP and port number, such as 10.0.0.1-9090.
  3. Delete the data files in that directory, such as all files under the data/10.0.0.1-9090/chunks directory.
  4. Restart the Prometheus service to reload the configuration files and data.

If Prometheus still has issues, you can try clearing the Prometheus data directory and restarting the Prometheus service. If the problem persists, consider redeploying the Prometheus component.

| username: redgame | Original post link

If a TiKV node is marked as “tombstone,” it may indicate that the node has been successfully scaled down and marked for deletion, but the data deletion process is not yet complete.

| username: DBRE | Original post link

This is the number of items historically in the tombstone state, which can be ignored.

| username: zhimadi | Original post link

Are there any official documentation recommendations to follow?

| username: zhimadi | Original post link

It shouldn’t be. The data was deleted a long time ago. Even the instance was released.

| username: zhimadi | Original post link

So it’s normal data display? Does it always exist and represent the historical number of tombstones?

| username: xfworld | Original post link

According to the official handling method, first scale down Prometheus, then scale up a new one, and it will be fine… :upside_down_face: :upside_down_face: :upside_down_face:

| username: DBRE | Original post link

It’s not scientific, is it? The data source for Prometheus is the PD API. If you scale down Prometheus and then scale it up again, will the data from PD change?

| username: xfworld | Original post link

Read more documentation, practice more, and your understanding will naturally improve…

| username: DBRE | Original post link

Can you help explain the mechanism behind it?

| username: 像风一样的男子 | Original post link

Use pd-ctl -u http://pd_ip:2379 store remove-tombstone to clean it up.

| username: xfworld | Original post link

Well, you’ll gain more by researching on your own :upside_down_face: Keep it up!

| username: DBRE | Original post link

In version 5.2.2, after scaling down Prometheus and then scaling it back up, the number of tombstones remains unchanged. :sweat_smile: :sweat_smile: :sweat_smile:

The method provided by tidb-feng: using pd-ctl -u http://pd_ip:2379 store remove-tombstone can reset the tombstone count to zero.

| username: DBRE | Original post link

The meaning of this value was previously confirmed with the official team by our colleague.

| username: 普罗米修斯 | Original post link

tiup ctl:v5.4.2 pd -u http://pd:2379 -i
store remove-tombstone
Remove

| username: xingzhenxiang | Original post link

You can handle it without the ctl tool, there’s no need to install the ctl tool.

curl -X DELETE {pd_leader_ip}:2379/pd/api/v1/stores/remove-tombstone

| username: zhimadi | Original post link

If it is normal data display and meaningful, keep it, do not clear it.

| username: zhimadi | Original post link

Thanks, I’ve heard it’s useful, so I won’t delete it for now.

| username: xingzhenxiang | Original post link

Tombstones are already meaningless for the cluster; the corresponding nodes no longer have any data and cannot be reused, so they should be cleared.