Monitoring shows that there are 2 TiKV Tombstone Stores in the TiDB cluster

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 监控显示tidb集群有2个tikv是Tombstone Stores

| username: zhanggame1

[TiDB Usage Environment] Production Environment / Testing / POC
[TiDB Version] 7.5
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
If Grafana monitoring shows this, is it a Grafana display issue or is it because the cluster has not properly cleaned up the previously scaled-in TiKV?

image
image720×450 21.1 KB

image
image1173×385 34.7 KB

2
| username: Fly-bird | Original post link

Did you take these 2 KV nodes offline? There should be one last step to do, check the documentation.

3
| username: zhanggame1 | Original post link

I did it, but tiup cluster display is not visible.

4
| username: 路在何chu | Original post link

The information of the two offline TiKV nodes should not have been deleted on the PD node.

5
| username: zhanggame1 | Original post link

How do I delete this?

6
| username: 普罗米修斯 | Original post link

Bro, did you forget to reply? You encountered this. Check the screenshot.

7
| username: Jjjjayson_zeng | Original post link

I’ve encountered this situation before; the server where the TiDB node is located has already become unresponsive.

8
| username: Jjjjayson_zeng | Original post link

You can use the above command to operate, otherwise, log in to the machine to kill the thread.

9
| username: zhanggame1 | Original post link

I just remembered, haha.

10
| username: zhanggame1 | Original post link

I checked the data in PD and can only see 3 TiKV instances. It still seems to be a monitoring issue.

» store
{
  "count": 3,
  "stores": [
    {
      "store": {
        "id": 2,
        "address": "10.10.10.207:20160",
        "version": "7.5.0",
        "peer_address": "10.10.10.207:20160",
        "status_address": "10.10.10.207:20180",
        "git_hash": "bd8a0aabd08fd77687f788e0b45858ccd3516e4d",
        "start_timestamp": 1702543376,
        "deploy_path": "/tidb-deploy/tikv-20160/bin",
        "last_heartbeat": 1704941589829448281,
        "state_name": "Up"
      },
      "status": {
        "capacity": "296.8GiB",
        "available": "173.3GiB",
        "used_size": "91.12GiB",
        "leader_count": 743,
        "leader_weight": 1,
        "leader_score": 743,
        "leader_size": 68598,
        "region_count": 2222,
        "region_weight": 1,
        "region_score": 482468.09045993385,
        "region_size": 206588,
        "slow_score": 1,
        "slow_trend": {
          "cause_value": 250030.06666666668,
          "cause_rate": 0,
          "result_value": 11,
          "result_rate": 0
        },
        "start_ts": "2023-12-14T16:42:56+08:00",
        "last_heartbeat_ts": "2024-01-11T10:53:09.829448281+08:00",
        "uptime": "666h10m13.829448281s"
      }
    },
    {
      "store": {
        "id": 7,
        "address": "10.10.10.209:20160",
        "version": "7.5.0",
        "peer_address": "10.10.10.209:20160",
        "status_address": "10.10.10.209:20180",
        "git_hash": "bd8a0aabd08fd77687f788e0b45858ccd3516e4d",
        "start_timestamp": 1702543289,
        "deploy_path": "/tidb-deploy/tikv-20160/bin",
        "last_heartbeat": 1704941590878681522,
        "state_name": "Up"
      },
      "status": {
        "capacity": "296.8GiB",
        "available": "169.6GiB",
        "used_size": "97.08GiB",
        "leader_count": 735,
        "leader_weight": 1,
        "leader_score": 735,
        "leader_size": 68643,
        "region_count": 2222,
        "region_weight": 1,
        "region_score": 485647.8300894471,
        "region_size": 206588,
        "slow_score": 1,
        "slow_trend": {
          "cause_value": 250046.5,
          "cause_rate": 0,
          "result_value": 4,
          "result_rate": 0
        },
        "start_ts": "2023-12-14T16:41:29+08:00",
        "last_heartbeat_ts": "2024-01-11T10:53:10.878681522+08:00",
        "uptime": "666h11m41.878681522s"
      }
    },
    {
      "store": {
        "id": 1,
        "address": "10.10.10.208:20160",
        "version": "7.5.0",
        "peer_address": "10.10.10.208:20160",
        "status_address": "10.10.10.208:20180",
        "git_hash": "bd8a0aabd08fd77687f788e0b45858ccd3516e4d",
        "start_timestamp": 1702572040,
        "deploy_path": "/tidb-deploy/tikv-20160/bin",
        "last_heartbeat": 1704941591831615481,
        "state_name": "Up"
      },
      "status": {
        "capacity": "296.8GiB",
        "available": "171.9GiB",
        "used_size": "95.82GiB",
        "leader_count": 744,
        "leader_weight": 1,
        "leader_score": 744,
        "leader_size": 69347,
        "region_count": 2222,
        "region_weight": 1,
        "region_score": 483825.71011686197,
        "region_size": 206588,
        "slow_score": 1,
        "slow_trend": {
          "cause_value": 250012.09166666667,
          "cause_rate": 0,
          "result_value": 33.5,
          "result_rate": 0
        },
        "start_ts": "2023-12-15T00:40:40+08:00",
        "last_heartbeat_ts": "2024-01-11T10:53:11.831615481+08:00",
        "uptime": "658h12m31.831615481s"
      }
    }
  ]
}
11
| username: 普罗米修斯 | Original post link

The tombstone mode is not displayed in pd-ctl. Execute the command and then refresh the monitoring to check.

12
| username: TiDB_C罗 | Original post link

I remember it showed “state_name”: “Down” status.
Executed store delete 14493, and then executed store remove-tombstone.

13
| username: WalterWj | Original post link

This is not an issue, it’s just that PD will process this offline information, so the monitoring will capture and display it. This is just to inform you of how many nodes have been scaled down historically. The methods mentioned above are for deleting the relevant information in PD.

14
| username: zhanggame1 | Original post link

It’s normal now.

15
| username: wangccsy | Original post link

How to modify it correctly.

16
| username: zhanggame1 | Original post link

Execute

tiup ctl:v7.5.0 pd -u http://10.10.10.211:2379 -i

Enter the console, and execute

store remove-tombstone
17
| username: dba远航 | Original post link

Execute the pruning command.

18
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.