After adding a machine online, the disk usage is abnormal. Can we stop migrating regions?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 线上挂了一台机器后,磁盘占用异常,可以停止迁移region吗

| username: zzw6776

【TiDB Usage Environment】Production Environment
Version 5.3
At 7:30 AM, one of the online machines suddenly went offline, and the disk usage increased abnormally.

image
image1378×186 31.1 KB

According to the monitoring, it was due to region migration, which caused the disk usage to increase abnormally. In this situation, queries are particularly slow. Is there a way to stop the region migration first to meet business needs?
image
image1242×398 106 KB

image
image1277×341 59.7 KB

2
| username: Christophe | Original post link

You can scale down this machine, replace the disk, and then scale it back up.

3
| username: zzw6776 | Original post link

I feel that it’s not a problem of scaling in or out, but rather that the scheduling of regions is occupying the disk.

4
| username: ddhe9527 | Original post link

You can temporarily lower the region-schedule-limit and replica-schedule-limit, and then increase them during off-peak business hours to quickly replenish the replicas.

5
| username: zzw6776 | Original post link

In the group, an expert @李启航 provided a solution:
pd-ctl store limit all 0

6
| username: ddhe9527 | Original post link

666, limit from the consumer end: +1:

7
| username: tidb狂热爱好者 | Original post link

This method is good, it will not automatically schedule anymore.

8
| username: ddhe9527 | Original post link

Actually, I think this configuration, although effective, is a bit too extreme. What about newly created regions? Wouldn’t it be more appropriate to set enable-replace-offline-replica to false?

9
| username: 西伯利亚狼 | Original post link

I think it’s also this problem.

10
| username: zzw6776 | Original post link

Final conclusion:
The disk usage is all pulled up by TiFlash.
Therefore, adjusting enable-replace-offline-replica and enable-remove-down-replica is useless because these two parameters cannot restrict TiFlash.
Restricting the store can directly restrict TiFlash.

11
| username: ddhe9527 | Original post link

Is it the TiFlash node that is down?

12
| username: zzw6776 | Original post link

The machine that crashed has both TiKV and TiFlash.

13
| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.