A TiKV node that has already been uninstalled recently started showing information in the monitoring system again. How can I completely remove it?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 已经卸载的一个tikv节点,最近又在监控中显示信息,请问如何彻底删除。

| username: TiDBer_Y2d2kiJh

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version] 2tidb 3pd 3tikv
[Reproduction Path] On August 15th, the IO of this tikv reached 100%. First, we stopped it, then scaled down, and finally cleaned up the tombstone stores. Recently, we found related information displayed in the monitoring again. How can we completely clear this tikv information?
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]


Abnormalstores

| username: 像风一样的男子 | Original post link

Clean up nodes in tombstone status in Grafana
pd-ctl -u http://0.0.0.0:2379 store remove-tombstone

| username: songxuecheng | Original post link

Reload Grafana again.

| username: tidb菜鸟一只 | Original post link

Try reloading Prometheus with tiup cluster reload -R prometheus.

| username: TiDBer_Y2d2kiJh | Original post link

Tried it, didn’t work. I guess we still need to clear the information in pd-ctl store.

| username: TiDBer_Y2d2kiJh | Original post link

Tried it, still doesn’t work. I guess we still need to clear the information in pd-ctl store.

| username: redgame | Original post link

pd-ctl -u http://0.0.0.0:2379 store remove-tombstone, this can be done.

| username: tidb菜鸟一只 | Original post link

No, you still have it in PD. I thought it was only cached in Prometheus. In that case, use pdctl to delete the node.

| username: TiDBer_Y2d2kiJh | Original post link

pd-ctl can only handle Tombstone Stores status, right?

| username: TiDBer_Y2d2kiJh | Original post link

Can the status of this node, which is Down Stores, also use remove-tombstone? This is the first time I’ve encountered this situation.

| username: Fly-bird | Original post link

How about trying to restart the cluster?

| username: h5n1 | Original post link

How did you scale down, did you use --force? Your monitoring status is down, check information_schema.tikv_store_status to see if you can see the scaled-down node.

| username: 像风一样的男子 | Original post link

Check the status of the store in pd-ctl to see if there are any residual regions in the down store.

| username: tidb菜鸟一只 | Original post link

Try deleting it directly. Why is it down? Haven’t you already scaled in this node with tiup?

| username: TiDBer_Y2d2kiJh | Original post link

It uses --force, you can see it in information_schema.tikv_store_status, it’s in the screenshot above.

| username: TiDBer_Y2d2kiJh | Original post link

The image you provided is not accessible. Please provide the text content you need translated.

| username: h5n1 | Original post link

Executing pd-ctl store delete xxx should change the status to offline, and then wait for the leader region count to become 0.

If it remains unchanged for a long time, refer to the document below for handling:

| username: 像风一样的男子 | Original post link

Try the three-step method mentioned above. Generally, KV is in an available state. Avoid using --force to shrink nodes as it can be troublesome to handle later.

| username: TiDBer_Y2d2kiJh | Original post link

The expert wrote in great detail, but I haven’t operated it before. Does xxx represent store_id?

| username: h5n1 | Original post link

Yes.