Abnormal Empty Region

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 异常空region

| username: dbaspace

[TiDB Usage Environment] Production Environment
[TiDB Version] v4.0.10
[Reproduction Path] Operations that led to the issue

  1. Performed unsafe downscale operations on 2 TiKV nodes
  2. After deleting the region cache, the issue persisted
    image

[Encountered Issue: Symptoms and Impact]
Cluster write operations are slow, DM to TiDB write operations are abnormal, with error message Error 9005: Region is unavailable. There are 10 regions without a leader, as shown in the image

image
image1380×644 70.9 KB

Other monitoring

image
image1380×242 50.1 KB

2
| username: xfworld | Original post link
  1. First, check the status of the region to see if it meets the replica configuration requirements.
  2. Check if the number of surviving region leaders is correct…
  3. For released regions, enhance the release scheduling (accelerate region removal).
  4. If there are abnormal regions, consider using tikv-cli commands to handle them… (provided that the region metadata in PD is normal).
3
| username: TiDBer_jYQINSnf | Original post link

It is estimated that these 10 regions need to perform unsafe recovery. First, check if all replicas of these 10 regions have 2 replicas in the TiKV you took offline. If so, use unsafe recovery to restore them. Data loss may occur.

4
| username: 小龙虾爱大龙虾 | Original post link

Isn’t the region metadata in PD reported by the regions on the TiKV nodes?

5
| username: xfworld | Original post link

After performing the unsafe offline operation on 2 TiKV nodes, the regions became inconsistent.

6
| username: 小于同学 | Original post link

Bug

7
| username: dbaspace | Original post link

Yes, the cluster will recover after fixing several abnormal regions.

8
| username: IanWong | Original post link

The v4 version is too old and will reach EOL on 2024-04-02. It is recommended to schedule an upgrade.

9
| username: h5n1 | Original post link

If there is a region without a leader, recreate it.

10
| username: TiDBer_rvITcue9 | Original post link

Recreate

11
| username: 呢莫不爱吃鱼 | Original post link

Recommend upgrading

12
| username: zhaokede | Original post link

:+1:

13
| username: Hacker_PtIIxHC1 | Original post link

:+1: :+1:

14
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.