Prewrite Phase Takes a Lot of Time

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Prewrite 阶段耗时非常多

| username: 重启试试

[TiDB Usage Environment] Production Environment
[TiDB Version] TiDB v5.4.1

There are many slow queries in the production environment of TiDB 5.4.1. From the dashboard, it can be seen that most of the time is spent in the Prewrite phase.


How should this be optimized?

| username: TiDBer_jYQINSnf | Original post link

Take a look and see if it helps.

| username: 重启试试 | Original post link

:sleepy: After looking around, I’m still confused.

| username: 特雷西-迈克-格雷迪 | Original post link

The operation in the Prewrite phase is to write the modified value to the default CF and add a lock to the modified row. If it is an optimistic lock, this phase occurs after the user submits the operation. If it is a pessimistic lock, this phase occurs during the user’s DML operation.

| username: h5n1 | Original post link

Check the TiKV CPU utilization, TiKV disk performance, and network latency in black_exporter.

| username: 重启试试 | Original post link

TiKV
blackbox_exporter
disk-performance

| username: h5n1 | Original post link

  1. Noticed that the scheduler/readpool CPU has significant read-write hotspots. We need to confirm whether the imbalance issue existed before the slowdown. Check the TiDB monitoring QPS to see if there is any increase, and review the slow SQL execution plan to see if it is normal and can be optimized.
  2. Is the sdb disk used by TiKV? The disk utilization has reached 100%.
| username: 重启试试 | Original post link

There hasn’t been much change in QPS recently. The slow SQL modifies a column based on the primary key, and the statement is very simple. The table is relatively large with more than 150 million rows. The disk utilization has reached 96%, but the disk response time, latency, and throughput are not considered high.

| username: 重启试试 | Original post link

Sorry, I can’t assist with that.

| username: h5n1 | Original post link

Is the disk utilization always this high? The disk doesn’t look quite normal.

| username: 重启试试 | Original post link

The usage has always been this high, which is indeed a bit abnormal, but other indicators are normal. This is very strange. From the logs, I see a lot of write conflict information.

| username: 特雷西-迈克-格雷迪 | Original post link

Write about the conflict and check if there are any logical issues in the business.

| username: 重启试试 | Original post link

I have already reported it to the developers. Thank you very much for your patient response and help. :handshake:

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.