Pending Task Alert in TiDB

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb的报警pending_task

| username: tidb狂热爱好者

【TiDB Usage Environment】Production Environment / Testing / PoC
【TiDB Version】
【Encountered Issues】
【Reproduction Path】What operations were performed to encounter the issue
【Issue Phenomenon and Impact】
How to handle task accumulation

[FIRING:1] TiKV_pending_task

Labels

  • alertname: TiKV_pending_task
  • cluster: bib_tidb_prod
  • env: bib_tidb_prod
  • expr: sum(tikv_worker_pending_task_total) BY (instance,name) > 1000
  • instance: 10.108.33.4:20181
  • level: warning
  • monitor: prometheus
  • name: gc-worker

Annotations

  • description: cluster: bib_tidb_prod, type: , instance: 10.108.33.4:20181, values: 3487
  • summary: TiKV pending task too much
  • value: 3487

Source: http://10.108.33.140:9090/graph?g0.expr=sum+by(instance%2C+name)+(tikv_worker_pending_task_total)+>+1000&g0.tab=1
【Attachments】

Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.

2
| username: Kongdom | Original post link

Refer to this link for a check:

3
| username: xiaohetao | Original post link

Check the values of the following monitoring items:

  • Scheduler pending commands: The ops of pending commands on each TiKV instance
  • Compaction pending bytes: The size of data waiting for compaction
  • Worker pending tasks: The number of pending and running tasks per second in the current worker. Normally, it should be less than 1000