Eliminating PD_down_peer_region_nums Alert

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: PD_down_peer_region_nums告警消除

| username: TiDBer_pkQ5q1l0

[TiDB Usage Environment] Production Environment
[TiDB Version] V6.1.0
[Encountered Issue] PD_down_peer_region_nums Alert
[Issue Phenomenon and Impact]
The alertmanager shows a PD_down_peer_region_nums alert. How can I fix it?

1662602700781
1662602700781854×224 83.1 KB

2
| username: Billmay表妹 | Original post link

Could it be caused by this issue~

3
| username: Billmay表妹 | Original post link

PD_down_peer_region_nums

  • Alert Rule: (sum(pd_regions_status{type="down-peer-region-count"}) by (instance) > 0) and (sum(etcd_server_is_leader) by (instance) > 0)
  • Rule Description: The number of Regions reported by the Raft leader that have unresponsive peers.
  • Handling Method:
    • Check if there is any TiKV downtime, recent restart, or if it is busy.
    • Observe the region health panel and check if the down_peer_region_count is continuously decreasing.
    • Check for network connectivity issues between TiKVs.
4
| username: songxuecheng | Original post link

pd-ctl region check down-peer

5
| username: TiDBer_pkQ5q1l0 | Original post link

How should I handle the check result? down.pd (33.0 KB)

6
| username: songxuecheng | Original post link

Have there been any previous operations? Check if there are any issues with these three stores.

7
| username: TiDBer_pkQ5q1l0 | Original post link

Expanded by 3 TiKV nodes.

8
| username: songxuecheng | Original post link

If it doesn’t keep increasing, you can wait for the cluster expansion balance to complete and then observe.

9
| username: songxuecheng | Original post link

This node with “store_id”: 5 is a bit strange, are there any other operations?

10
| username: TiDBer_pkQ5q1l0 | Original post link

The expansion was done yesterday, and today I see that the balance has been completed. There were no other operations besides the expansion.

11
| username: songxuecheng | Original post link

You can continue to observe; it seems to still be doing region balance.

12
| username: TiDBer_pkQ5q1l0 | Original post link

Okay, thank you.

13
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.