Upgrade Check Reports Some Regions Unhealthy

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 升级检查报错部分region 不健康

| username: DBAER

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issues: Issue Phenomenon and Impact]
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]

  1. Preparing for cluster upgrade, directly checked and found some regions in the cluster are abnormal.
  2. Checked the monitoring and found some regions are down and some are pending.
  3. Checked the monitoring and found logs near the real-time point.
    image
    image1380×590 393 KB
  4. Checked the nodes, all are up, no restart actions occurred.
  5. Checked the forum, similar cases require manual removal of errors, not sure if this is the correct handling method.

image
image1380×414 75.4 KB

2
| username: TiDBer_jYQINSnf | Original post link

From pd-ctl, check region check miss-peer to see if there are any regions missing the majority of replicas. If not, then it’s fine. These logs just indicate that the merge cannot be executed.

3
| username: DBAER | Original post link

The issue has been resolved. It was caused by a node’s disk space exceeding 95%, leading to abnormal behavior.

4
| username: 小龙虾爱大龙虾 | Original post link

Can you tell me how your disk is full with just 2.3 GB of data?

5
| username: DBAER | Original post link

Test reuse environment

6
| username: TiDBer_JUi6UvZm | Original post link

Excellent :+1:

7
| username: zhanggame1 | Original post link

When the usage rate reaches 80%, data should no longer be written.

8
| username: TiDBer_QYr0vohO | Original post link

:+1: