After scaling down the tidb-server node in TiDB, the monitoring shows errors

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb的tidb-server节点缩容后，监控显示错误

| username: TiDBer_lei_xin

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Problem Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

After scaling down a tidb-server node in TiDB 5.3.3, the monitoring panel shows errors.
Before scaling down:

image
image1380×546 201 KB

After scaling down:

image
image1380×562 215 KB

Monitoring panel:

image
image1380×366 22.2 KB

2
| username: xfworld | Original post link

Everything seems normal.

Try this command:

3
| username: 裤衩儿飞上天 | Original post link

Try reloading Prometheus and Grafana to see if the three down instances disappear.

4
| username: Kongdom | Original post link

You should restart the cluster, it seems like there is some residual information.

5
| username: tidb菜鸟一只 | Original post link

tiup cluster reload web_ind_cds_tidb -R prometheus

6
| username: xingzhenxiang | Original post link

Will it be fine after a while?

7
| username: TiDBer_lei_xin | Original post link

I restarted the Prometheus node, but it still hasn’t disappeared.

8
| username: TiDBer_lei_xin | Original post link

I have executed this command, but it still doesn’t seem to work.

9
| username: TiDBer_lei_xin | Original post link

Prometheus, alert diagram:

image
image1378×316 17 KB

10
| username: TiDBer_lei_xin | Original post link

Thank you very much. After executing
tiup cluster reload web_ind_cds_tidb -R prometheus
the alert disappeared.

11
| username: 裤衩儿飞上天 | Original post link

reload is not restart

12
| username: TiDBer_lei_xin | Original post link

Yes, using reload has solved the issue now. Thank you very much. I have scaled down several clusters without any problems, but why does only this cluster have this issue?

13
| username: 裤衩儿飞上天 | Original post link

If it’s a metaphysical issue, can you accept it? :crazy_face:

14
| username: TiDBer_lei_xin | Original post link

Haha, I can accept that.

15
| username: liuis | Original post link

Restarting is a good method.