Write Data IO 100%

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 写入数据 io100%

| username: 宸凡_22

When writing data, IO performance reaches 100%, while CPU and memory usage are very low, as shown in the picture:

io100%25
io100%251380×868 103 KB

How can I optimize this?

2
| username: buddyyuan | Original post link

What disk? Check the Disk-Performance graph in Grafana. Look at the disk latency, IOPS, and bandwidth.

3
| username: TiDBer_CEVsub | Original post link
  1. You can optimize single insert statements into small batch insert statements, which will reduce the number of transaction commits and theoretically improve performance.
  2. Replace the disk, as the disk read/write performance is too low.
4
| username: Minorli-PingCAP | Original post link

The IO utilization metric is important, but a 100% utilization does not necessarily indicate a performance issue. It needs to be considered together with latency and throughput. On many SSDs, 100% utilization does not show any problems, but on SAS SATA type disks, this metric needs attention.

5
| username: db_user | Original post link

What kind of disk? I have a cloud io2 type disk here. This issue has occurred before, but usage only indicates that the disk is being read or written to at the moment, not that the disk is busy. You can determine if the disk is truly busy by checking the I/O latency.

6
| username: 宸凡_22 | Original post link

SSD Disk

disk performance data
disk performance data1380×610 576 KB

7
| username: 宸凡_22 | Original post link

First point: feasible; the disk is SSD.

8
| username: 宸凡_22 | Original post link

The queue for writing data is piling up.

9
| username: tidb狂热爱好者 | Original post link

I see your TPS is only 1500, it should be cloud SSD, like Alibaba virtual machines. Using this kind of SSD with TiDB is a bit strenuous.

10
| username: 宸凡_22 | Original post link

Using SSD disks, self-built virtual machine cluster, not the supplier’s virtual machine.

11
| username: tidb狂热爱好者 | Original post link

You should test the IO. The IO of the self-built virtual machine does not meet the standard. The SSD read/write speed should be around 3000 MB/s.

12
| username: tidb狂热爱好者 | Original post link

It is best to deploy on bare metal. The monitoring screenshot you provided is clearly incorrect; the number of changes per second is only 1500, and it’s already maxed out. A normal SSD should handle 300,000.

13
| username: 宸凡_22 | Original post link

For SSD disk IO testing, the write speed can reach around 200,000.

14
| username: 宸凡_22 | Original post link

The monitoring end shows around 10k.

15
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.