Unable to Retrieve Values in Prometheus Metric

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Prometheus Metric 中取不到值

| username: pupillord

【TiDB Usage Environment】Production Environment
【TiDB Version】V5.1.4 ARM
【Encountered Issues】
Unable to retrieve data through PQL queries in Prometheus
Unable to obtain corresponding data when querying various components of the cluster using probe_success
【Reproduction Path】Operations performed that led to the issue
【Issue Phenomenon and Impact】
Below is the test environment (displayed normally):
When I use probe_success{group=“tidb”} to query all TiDB components in the cluster, I can retrieve the data. There are three records, and the group is tidb for all of them.

image
image1380×548 26.5 KB

Unable to get the image in the production environment, but the phenomenon is:
Using probe_success{group=“tidb”} does not retrieve any data, and replacing tidb with other components also does not retrieve data.
Using probe_success{} without group filtering shows data, but none of them have the group field.

Questions:

  1. Currently, only one set of databases has this issue. What could be the reason?
  2. How to troubleshoot this issue, and what is the troubleshooting approach?
  3. Where can I find the configuration information for the probe_success metric?
2
| username: Min_Chen | Original post link

probe_success is a metric generated by Prometheus. When data is pulled from an instance, it is marked as 1. The configuration information is in the Prometheus configuration file under the configuration item with the job_name suffix “probe”.

For troubleshooting, it is recommended to understand the entire process of Prometheus scraping data. Generally, probe checks whether the TCP port can be connected.

3
| username: pupillord | Original post link

This issue has been investigated and resolved. The reason was that the corresponding replacement blackbox exporter and node exporter nodes had crashed. The root cause was that we had previously configured ignore_exporter: true, and after stopping the blackbox exporter and node exporter, they were not restarted. :joy:

4
| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.