TiKV Monitoring Lost

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv监控丢失

| username: julyxiong

【TiDB Usage Environment】Production
【TiDB Version】v3.0.3
【Encountered Problem】TiKV monitoring lost
【Reproduction Path】
【Problem Phenomenon and Impact】
The monitoring of one TiKV in Prometheus is lost, and another one has a monitoring breakpoint.

131DC61F-272E-4262-94E5-0567E0EAB619
131DC61F-272E-4262-94E5-0567E0EAB6191380×338 51.8 KB

2
2632×722 51.8 KB

2
| username: julyxiong | Original post link

Can this issue be resolved by directly restarting TiKV?

3
| username: xfworld | Original post link

The issues need to be looked at separately:

  1. Is the TiKV node service functioning normally?
  2. Are the black box_ exporter and Node Exporter services on the TiKV node functioning normally?

If the TiKV service is functioning normally, restarting the TiKV service won’t help in starting the exporter…

Please refer to this~

4
| username: Kongdom | Original post link

Could it be caused by the repeated switching of the PD leader?

5
| username: wuxiangdong | Original post link

First, check if the ports for TiKV and the exporter are still available.

6
| username: julyxiong | Original post link

Unable to wait for community assistance, I directly restarted the TiKV instance :sweat_smile:

After the restart, the metrics interface of TiKV is back to normal.

Now waiting for the TiKV Scheduler error to return to normal…