After expanding tikv-server, Grafana does not monitor information of the latest nodes

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv-server扩容后,grafna监控不到最新节点的信息

| username: TiDBer_FMWXa7ja

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version] 6.5
[Reproduction Path]
[Encountered Problem: Problem Phenomenon and Impact] Initially, there were three KV nodes deployed on servers 155, 156, and 157. Later, an additional KV-server node was added to each machine. However, Grafana can only monitor the information of the original three nodes and cannot monitor the information of the newly added three nodes.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

  1. Configuration files of the newly added three nodes

  2. Cluster information

  3. Grafana monitoring page image, only showing the KV-server information of port 20180 on the three machines, without the information of port 20181

| username: TiDBer_ZsnVPQB4 | Original post link

Log in to the newly added nodes and check if the node_exporter-9000 and blackbox_exporter-9115 services are running properly and if the ports are in listening state.

For CentOS 7 or Red Hat 7, use the following commands:

systemctl list-unit-files | grep node_
systemctl list-unit-files | grep black
systemctl status node_exporter-9100.service
systemctl status blackbox_exporter-9115.service
netstat -nltp

Also, ensure that the security group and firewall have the corresponding ports open. In any case, make sure these two services are running properly and the port network connections are normal.

| username: WalterWj | Original post link

Try manually reloading Prometheus with tiup.
tiup cluster reload c-name -R prometheus;

| username: Kongdom | Original post link

Try restarting the monitoring node.

| username: tidb菜鸟一只 | Original post link

Restart Prometheus

| username: TiDBer_FMWXa7ja | Original post link

My cluster has been restarted, and Prometheus has also been restarted, but it has no effect.

| username: TiDBer_FMWXa7ja | Original post link

These two services are normal, and the other three kv-server nodes can also be monitored.

| username: buddyyuan | Original post link

You need to check the Prometheus configuration on the Prometheus node to see if your expansion to TiKV is included in the configuration. Normally, you need to reload Prometheus, which will regenerate the Prometheus configuration through the mete.yaml file and then distribute it to remote nodes to overwrite the configuration. Follow this approach to investigate.

| username: TiDBer_FMWXa7ja | Original post link

Thank you, this command worked.

| username: Kongdom | Original post link

Understood, please provide the Chinese text you need translated.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.