TIKV apply wait is particularly high

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIKV apply wait特别高

| username: heeewei

[TiDB Usage Environment] Production Environment
[TiDB Version] 7.1.0
Using TiSpark to perform two-phase commit to write into TiKV, encountering leader drop. Investigation found that TiKV apply wait is particularly high, but the CPU usage of the apply thread is not high, as shown in the figure below. Could you please advise what might be causing the high apply wait?

Thread pool settings are as follows:
raftstore.apply-pool-size=5
raftstore.store-pool-size=4

| username: xfworld | Original post link

Reference documentation: 延迟的拆解分析 | PingCAP 文档中心

It is recommended to check whether the write operations of all TiKV nodes in the cluster are normal:

  • Apply wait duration: Histogram of apply wait times
  • Apply wait duration per server: Histogram of apply wait times for each TiKV instance

Then check the following two points:

Reference documentation: TiDB 集群问题导图 | PingCAP 文档中心

| username: heeewei | Original post link

I have checked all of these. Both the raftstore and apply thread pools are normal. I also looked at individual apply threads and found no issues. I suspect that the apply wait is not waiting for the thread pool, but for something else (rate limiting or locks?). Do you have any relevant troubleshooting ideas?

| username: xfworld | Original post link

There is no more information, so it’s up to you to make the judgment.

| username: redgame | Original post link

Is there a phenomenon where the disk write speed of TiKV nodes is relatively slow? Check the disk IOPS.

| username: magongyong | Original post link

Have you found the reason? Could you share it with us? We are facing the same issue and are about to change servers.

| username: in-han | Original post link

I also encountered this problem, with the same phenomenon.
The CPU usage for apply is not high, the delay for apply log/commit log is not high, but the apply wait is very high.