Replacing Monitoring Nodes Without Collecting Metrics

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 替换监控节点不采集监控

| username: 18515065291

[TiDB Usage Environment] Production Environment
[TiDB Version]: 5.2.2, 5.2.3
[Encountered Issue]: Replacing Monitoring Machine
<1> Take the existing monitoring nodes offline, including Prometheus, Grafana, and Alertmanager.
<2> Then scale out to the same physical machine with the following configuration:

- host:
  port: 666
  deploy_dir: /work/tidb666/prometheus-777
  data_dir: /work/tidb666/prometheus-777/data
  log_dir: /work/tidb666/prometheus-777/log
  storage_retention: 5d

- host:
  port: 888
  deploy_dir: /work/tidb666/grafana-888

- host:
  web_port: 999
  cluster_port: 555
  deploy_dir: /work/tidb666/alertmanager-555
  data_dir: /work/tidb666/alertmanager-555/data
  log_dir: /work/tidb666/alertmanager-555/log

<3> After scaling out, it was found that the Prometheus monitoring item probe_success could not collect the three monitoring items of node_exporter, black_export, and grafana.
<4> Enabling the debug log level of black_exporter revealed that it does not collect the IP

| username: Min_Chen | Original post link


Check if these three monitoring items are configured in the Prometheus configuration file?

| username: srstack | Original post link

Try running tiup cluster reload -R prometheus.