Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: apply duration
[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.5
[Reproduction Path] Operations performed that led to the issue
TiDB write is slow, storage async write duration and apply duration are high
Questions:
- The cluster load is relatively low, CPU usage does not exceed 50%, and the related pool-size is basically set to 12, so CPU is not the bottleneck.
- What is the specific process of apply duration? Is it writing to level0 db?
- What are the common causes of slow apply duration (CPU reasons have been ruled out), and what are the improvement measures?
[Encountered Issues: Problem Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
The image is not visible. Please provide the text you need translated.
Currently, some instances are directly using NVMe SSDs, while others are using LVM (in serial mode) made up of 4 NVMe disks. However, there is no direct evidence indicating that the issue is with LVM.
There is nothing abnormal in the disk-performance interface.
The apply process is the process of applying the data written by the user from the raft db to the KV db, which is the actual process of persisting the user data to disk.
The main operation at this stage is writing data to the disk.
Therefore, the following troubleshooting ideas can be considered:
- Focus on analyzing which TiKV nodes have higher apply latency.
- Determine whether the latency is high across all nodes or only some nodes. Then, focus on analyzing the disk I/O situation.
- Look up relevant parameters on the official website and try adjusting the apply pool to see if it has any effect.
The troubleshooting ideas can also refer to this article.
The optimization of apply-pool-size can refer to the official introduction:
- The Apply thread pool is responsible for writing data to the disk. This configuration item specifies the number of threads in the Apply thread pool, i.e., the size of the Apply thread pool. When adjusting the size of the Apply thread pool, please refer to TiKV Thread Pool Tuning.
- Default value: 2
- Adjustable range: [1, CPU * 10]
Learning. Beginner level.
This process involves logging to disk and is related to TIKV writes. Check the IO and concurrency situation.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
|
username: magongyong
|
username: 
|
username: 
|
username: 
|
username: 
|
username: