Issues with Adjusting Prometheus Alert Thresholds

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: prometheus告警阀值调整问题

| username: 路在何chu

[TiDB Usage Environment] Production Environment
[TiDB Version]
4.0.13
[Reproduction Path] What operations were performed that caused the issue
Adjusted the monitoring increase(tikv_coprocessor_request_error{reason!=“lock”}[10m]) > 200 to greater than 200
[Encountered Issue: Problem Phenomenon and Impact]
After restarting the cluster, the monitoring is reset to the default value and needs to be modified again. Has anyone else encountered this?
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

| username: Fly-bird | Original post link

Is it because of your version issue? We haven’t encountered this problem, it can be modified.

| username: 路在何chu | Original post link

It can be modified, but after restarting the cluster, it reverts to the default value. You have to change it to 200 again and restart Prometheus.

| username: chenhanneu | Original post link

  • rule_dir: This field specifies a local directory that should contain complete *.rules.yml files. These files will be transferred to the target machine during the cluster configuration initialization phase and used as Prometheus rules.

Topology File Configuration for Deploying TiDB Cluster via TiUP | PingCAP Documentation Center

Put both the modified rules and the remaining unmodified ones into the rule_dir directory, and they will be automatically loaded after restarting.

| username: 路在何chu | Original post link

Okay, I’ll give it a try.

| username: 路在何chu | Original post link

The file was not found, and to make the changes to the alert threshold effective, Prometheus must be restarted. Reloading will revert to the initial configuration.

| username: 路在何chu | Original post link

If the rule_dir is not configured, it should first read the rule files in the bin directory, and then read the rule files in the conf directory.

| username: chenhanneu | Original post link

Move the required rule.yml file from the conf directory to your newly created rule_dir, and restart Prometheus. If this directory is not configured, the initial rule will be applied every time you reload.

| username: 路在何chu | Original post link

Okay, I’ll give it a try.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.