How to Redeploy Nodes for Alertmanager/Grafana/Prometheus That Have Gone Offline

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: alertmanager/grafana/prometheus部署的节点已下线，如何重新部署

| username: Queena

【TiDB Usage Environment】Production Environment
【TiDB Version】v5.2.4
【Encountered Problem】The node where alertmanager/grafana/prometheus was deployed is offline, how to redeploy
【Reproduction Path】Currently, display only shows PD, TiDB, and TiKV roles in the cluster. How to redeploy alertmanager/grafana/prometheus?

【Problem Phenomenon and Impact】
The alertmanager/grafana/prometheus previously deployed on one machine is now offline. How to redeploy alertmanager/grafana/prometheus on the existing nodes?

The node where alertmanager/grafana/prometheus was deployed is offline, how to redeploy

2
| username: ddhe9527 | Original post link

Scale down the offline alertmanager/grafana/prometheus services, and then scale them up again.

3
| username: Queena | Original post link

The scaling down has been completed. How do I scale up?
Use the command tiup cluster scale-out <cluster-name> scale-out.yaml to scale up. Now it reports an error: Error: add local rule: read local directory /export/servers/data/tidb/prometheus/deploy/ failed: open /export/servers/data/tidb/prometheus/deploy/: no such file or directory

4
| username: 啦啦啦啦啦 | Original post link

The method is the same as expanding other nodes. Write the topology configuration file and use tiup to expand.

5
| username: Queena | Original post link

The current error is: Error: add local rule: read local directory /export/servers/data/tidb/prometheus/deploy/ failed: open /export/servers/data/tidb/prometheus/deploy/: no such file or directory
But the directory does exist.

6
| username: ddhe9527 | Original post link

Check if the tidb user on the target machine has sufficient permissions to access and read/write to the directory. You can also refer to the tiup-cluster-debug logs for more details.

7
| username: luqiuhua | Original post link

As far as I know, the exporter might not have been deployed. Are the three monitoring nodes mixed with other TiDB nodes? Could you share the display and scale-out configurations?

8
| username: cs58_dba | Original post link

In theory, clearing the original setup and redeploying with tiup should be sufficient.

9
| username: Queena | Original post link

Yes, I mixed them. I modified the configuration file and it worked. Thank you.

10
| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.