The IO usage of TiDB is always high, but the actual disk IO is not high, seeking solution

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIDB的IO使用率总是很高，但实际磁盘IO并不高，求解

| username: 奋斗的大象

【TiDB Usage Environment】Production Environment
【TiDB Version】6.10
【Reproduction Path】What operations were performed when the issue occurred
【Encountered Issue: Issue Phenomenon and Impact】
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】
The IO on the TiDB console is often maxed out

image
image1114×334 132 KB

But the backend disk monitoring shows read/write speeds below 100M/S
image
image749×457 6.49 KB

Can any expert analyze the reason?

2
| username: tidb狂热爱好者 | Original post link

IOPS has already reached 2000, which is the bottleneck for mechanical drives. Read/write speed is 25MB/s.

3
| username: 啦啦啦啦啦 | Original post link

Is it an NVME drive?

4
| username: tidb狂热爱好者 | Original post link

Even if you make everything direct pass-through, the performance will be much better.

5
| username: Kongdom | Original post link

Is it the node machine corresponding to 100% IO?

6
| username: 像风一样的男子 | Original post link

The disk performance is a bit poor. If you switch to SSD, it will be like mine, fully utilized but very stable.

image
image1003×242 24.4 KB

7
| username: TiDBer_QKDdYGfz | Original post link

Can iostat be used?

8
| username: lemonade010 | Original post link

Print a screenshot of iostat to see if there is any wait. We are looking at TPS, not MB/S.

9
| username: zhanggame1 | Original post link

Is it a mechanical disk? Check the disk latency.

10
| username: zhaokede | Original post link

It is best to switch to NVME disks.

11
| username: WinterLiu | Original post link

Using iostat -x 30 to look at the last column, you can see the actual busy level of the disk.

12
| username: tony5413 | Original post link

What data disk are you using? Do you have a comparison of specific node time periods?

13
| username: tidb狂热爱好者 | Original post link

It seems that the disk is too poor.

14
| username: YuchongXU | Original post link

It’s okay to replace the disk.

15
| username: tidb狂热爱好者 | Original post link

Just replace the disk.

16
| username: TiDBer_xTvoCh2f | Original post link

My virtual machine’s disk is like this too.

17
| username: jiayou64 | Original post link

If the front-end service is affected, switch to a high-performance server.

18
| username: madcoder | Original post link
  1. Virtual machines will share disk performance, and some virtual machines may also limit read and write operations. 2. Mechanical disks themselves have very low read and write speeds, consider using a disk array.
19
| username: tidb狂热爱好者 | Original post link

You are using a mechanical hard drive.

20
| username: TiDBer_pOZcnpJA | Original post link

Take a look at what TiDB’s monitoring interface uses to determine IO load, then compare what you are looking at with TiDB’s data collection rules to see if they refer to the same thing.