Slow Prewrite in INSERT Statement

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: INSERT语句prewrite慢

| username: sgjr

【TiDB Usage Environment】Production Environment
【TiDB Version】4.0.14
【Reproduction Path】Operations performed that led to the issue
【Encountered Issue: Phenomenon and Impact】

  • Simple insert statements are executing slowly. Heatmap shows no write hotspots, and TiKV nodes are load balanced.
    【Resource Configuration】TiKV 9C/90G * 6
    【Attachments: Screenshots/Logs/Monitoring】

image
image1380×664 78.4 KB

2
| username: WalterWj | Original post link

:thinking: Is there a binlog?

3
| username: sgjr | Original post link

The TiDB binlog is not enabled.

4
| username: WalterWj | Original post link

Then check the disk performance monitoring to see if there are any high latency issues with TiKV IO.

5
| username: sgjr | Original post link

As shown in the images, the ioawait of the TiKV disk is not high, and the iops can reach up to 2k+. Since the disk is SSD, it should not have reached the disk IO bottleneck.

6
| username: Lucien-卢西恩 | Original post link

TiDB → KV Error → backoff/transaction and TiKV-details → Scheduler monitoring to see if there are any write conflict issues;

7
| username: sgjr | Original post link

Write conflicts also do not have exceptions. It has been found that the wait_time.snapshot of one TiKV node is often very high, sometimes lasting for several seconds. Later, the same issue occurred with updates in the same cluster. The logs are posted in another question: tikv wait_time.snapshot耗时很高 - #4，来自 tug_twf - TiDB 的问答社区