Is there a parameter in the TiKV configuration file to control disk write bandwidth in version V5.0.3?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 请问在V5.0.3版本下TiKV的配置文件参数中有控制磁盘写入带宽的参数吗

| username: Leox

As mentioned, is there a parameter in the TiKV configuration file under version V5.0.3 that controls disk write bandwidth?

Sysbench insertion of 5×100 million data always fails.

1676348216563
16763482165631513×169 14 KB

Because the disk is relatively strong, I am a bit worried that the disk is getting full and the CPU can’t handle it. (Just want to try it out.)

2
| username: Kongdom | Original post link

You can refer to this thread for tuning:

There is a bandwidth-related configuration in TiKV’s settings, but I haven’t used it before.
image

3
| username: Leox | Original post link

I also only found this one related to bandwidth in the configuration file :rofl:, it doesn’t seem to be what I want. I’m looking at the one above, thanks!

4
| username: Meditator | Original post link

Rate limiting can be divided into several aspects:

  1. On the tidb-server side, set the number of tokens to limit the number of connections on the frontend;
  2. Various thread rate limiting, the thread pool mainly consists of gRPC, Scheduler, UnifyReadPool, Raftstore, Apply, RocksDB, but mainly the Raftstore thread pool configuration, thereby limiting the number of CPU cores that write requests can use. In addition, there is also read request rate limiting;
  3. RocksDB rate limiting, by limiting the memtable size and the number of files at each level of the LSM, but this is terminal rate limiting and has little effect, often occurring in scenarios where CPU usage is already high; this mainly limits IO;
  4. Of course, there are also various memory configurations to limit the rate.
    Personally, I think the main adjustment should be the number of threads in the thread pool.
5
| username: Leox | Original post link

May I ask if it is normal to see the Scheduler worker CPU of one Tikv node in Grafana being much higher than the other two nodes? Additionally, after I have adjusted the storage.scheduler-worker-pool-size to 8, what does this red line in Grafana represent?

6
| username: ealam_小羽 | Original post link

Far above is definitely abnormal. The red line in Grafana represents the warning line. For example, if you think the SQL response time should be well below 500ms, you can set 500ms as the warning line. Anything exceeding 500ms is definitely abnormal and needs to be addressed.

Image
Image963×424 91.5 KB

7
| username: Leox | Original post link

Yes, but if my storage.scheduler-worker-pool-size is set to 8, why does this line appear near 100%? :rofl:

8
| username: ealam_小羽 | Original post link

It looks like it should be this issue:

Image
Image1106×474 138 KB

9
| username: Leox | Original post link

It doesn’t seem to be the case. Increasing the value of the scheduler-worker-pool-size parameter didn’t have any effect. It was only after I reduced the memory usage of TiKV during insertion and the number of threads used by sysbench that I successfully prevented TiKV from crashing during data insertion. :rofl:

10
| username: ealam_小羽 | Original post link

I thought your memory was fine, but it should be that the batch size is too large. The one above is somewhat transaction-oriented.

11
| username: Kongdom | Original post link

Is it a disk IO issue?

12
| username: Leox | Original post link

I think it’s very likely because one direct change caused by these operations is a reduction in disk I/O usage.

13
| username: Leox | Original post link

If there are too many batches, is there any good solution? Reducing the number of threads during insertion is too time-consuming :rofl:
raftstore.apply-max-batch-size / raftstore.store-max-batch-size might be effective?

14
| username: ealam_小羽 | Original post link

I don’t know how you use sysbench to generate data over there. If it’s the default method of generating data one row at a time with multiple threads, batch processing might be better?

15
| username: Leox | Original post link

Got it, thank you!