CDC Monitoring Dashboard Shows No Data

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: cdc监控面板显示no data

| username: HACK

【TiDB Usage Environment】Production, Testing, Research
【TiDB Version】

【Encountered Problem】
Yesterday, I manually stopped the changefeed. This morning, when I checked the CDC-related monitoring content on the Grafana interface, most of the panels showed “no data” and I couldn’t see any replication information. Is this an issue? It feels like it shouldn’t display “no data.”

Additionally:
After I restarted the changefeed that I stopped yesterday, the monitoring content was displayed.

【Reproduction Path】What operations were performed to encounter the problem
【Problem Phenomenon and Impact】

【Attachments】

Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.

2
| username: songxuecheng | Original post link
  1. Can ctl see information related to changefeed?
  2. Check what related operations were done at the earliest time the monitoring stopped.
  3. Firewall
  4. Try restarting Prometheus.
3
| username: HACK | Original post link

I don’t see any issues with ctl.
Yesterday, I started an insert operation upstream, continuously inserting data, then stopped the change feed. The next day, I restarted the change feed to check the related latency display on the Grafana monitoring interface.
But when I logged into Grafana this morning, the panel showed “no data,” and re-logging didn’t help.

Try restarting Prometheus: I haven’t tried this, but under normal circumstances, it shouldn’t display “no data.”

After I restarted the change feed, the panel displayed content.

4
| username: songxuecheng | Original post link

The change feed has never started, so it definitely doesn’t exist.

5
| username: HACK | Original post link

If the changefeed replication fails due to other reasons, will it also show “no data”? I feel this is not very user-friendly.

6
| username: songxuecheng | Original post link

Do you have only one changefeed?

7
| username: HACK | Original post link

Well, there’s only one.

8
| username: songxuecheng | Original post link

If only one changefeed is stopped, the corresponding monitoring will have no data. If there are multiple changefeeds, there will still be other data.

9
| username: HACK | Original post link

Well, I feel that as long as there is one changefeed, it would be good to still have content displayed. It might also be related to how Grafana itself retrieves and displays the information.

10
| username: jansu-dev | Original post link

The TiDB product system uses the Prometheus client for encapsulation, writes monitoring data to the Prometheus server, and then displays it using Grafana. If you disable CDC capture, it means that monitoring data will not be collected, resulting in no data later on.

This is akin to shutting down the database process while still expecting to provide database functionality.

11
| username: HACK | Original post link

Got it.

12
| username: system | Original post link

This topic will be automatically closed 60 days after the last reply. No new replies are allowed.