After expanding TiFlash, Grafana shows down

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 扩容tiflash后，grafana上显示down

| username: Christophe

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Problem Phenomenon and Impact]
After expanding TiFlash, Grafana shows down, but display and node processes are normal
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

image
image945×253 6.2 KB

image
image1202×20 2.17 KB

image
image1247×196 12.5 KB

2
| username: Kongdom | Original post link

Try restarting the Grafana node.

3
| username: Fly-bird | Original post link

Restart the monitoring or TiFlash node.

4
| username: xfworld | Original post link

Turn off the firewall…

5
| username: TiDBer_小阿飞 | Original post link

Restart the node

6
| username: xingzhenxiang | Original post link

Check the firewall to see if it is enabled.
On the Prometheus machine, run telnet xx.xx.xx.xx 9100
telnet xx.xx.xx.xx 9115

7
| username: ti-tiger | Original post link
  • Network Issues Between TiFlash and Prometheus: If the network between the TiFlash node and the Prometheus node is not functioning, Prometheus will be unable to retrieve monitoring data from TiFlash, causing Grafana to display the TiFlash node as down. You can try checking the network connectivity between the TiFlash node and the Prometheus node, such as using the ping or telnet command, or by checking the TiFlash logs for connection failure errors.
  • Port Issues Between TiFlash and Prometheus: If the ports between the TiFlash node and the Prometheus node are blocked by a firewall or other security policies, Prometheus will also be unable to retrieve monitoring data from TiFlash, causing Grafana to display the TiFlash node as down. You can try checking if the ports between the TiFlash node and the Prometheus node are open, such as using the netstat or iptables command, or by checking if the monitoring ports are correctly set in the TiFlash configuration file.
  • TiFlash Monitoring Component Issues: If the monitoring components of the TiFlash node (tiflash proxy and pd buddy) are malfunctioning or experiencing issues, it will also affect the reporting and display of TiFlash monitoring data, causing Grafana to display the TiFlash node as down. You can try checking the status and logs of the TiFlash node’s monitoring components, such as using the systemctl or journalctl command, or by reviewing the relevant metrics in the TiFlash monitoring documentation.