Write Latency Doubled After Upgrading from 6.1.2 to 6.5.2

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 6.1.2 升级 6.5.2 后写入耗时翻倍

| username: linw1995

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.2
[Reproduction Path] Upgrade from 6.1.2 to 6.5.2
[Encountered Problem: Phenomenon and Impact]
Before Upgrade

image
image1380×324 105 KB

After Upgrade
image
image1380×322 78.5 KB

[Resource Configuration]

image
image951×492 39 KB

2
| username: xfworld | Original post link

If the cluster configurations are exactly the same, it is best to find a detailed scenario for comparison, such as:

  • Same structure
  • Same number of rows and column information written
  • What the differences in writing are like
  • Additionally, you can supplement with a comparison of the write execution plans
  • You can also check for hardware differences in terms of IO
3
| username: xfworld | Original post link

This difference is quite significant…

4
| username: linw1995 | Original post link

The BatchGet of Scheduler is much slower than before the upgrade, but the Thread CPU doesn’t seem to be very high.

Before the upgrade:

image
image1209×268 87.8 KB

image
image1211×302 38.5 KB

After the upgrade:

image
image1208×265 73.2 KB

image
image1210×302 34.4 KB

5
| username: xfworld | Original post link

According to the direction mentioned in 读性能慢-总纲 - TiDB 的问答社区, check and see which part is the problem.

6
| username: linw1995 | Original post link

Thank you for providing the troubleshooting direction. Following the main outline, it was found that there is a phenomenon of extremely high long-tail latency in RocksDB’s Get operation.

Before the upgrade:

image
image1378×336 56.1 KB

After the upgrade:

image
image1380×334 50.7 KB

7
| username: xfworld | Original post link

The hit rate has decreased, and the query has slowed down…

You can refer to these metrics for further investigation:

However, I suggest finding a single table with relatively frequent business operations and try to fetch data through the primary key to see how long it takes to execute. Then, check the number of MVCC versions in the execution plan…

image
image764×247 23.4 KB

8
| username: linw1995 | Original post link

It shouldn’t be a load issue because performance dropped significantly after the upgrade, causing write timeouts. Subsequently, a large number of writes were discarded, and the insert QPS was halved.

The other cluster with dual writes is still on v6.1.2, and its performance has been stable with a low load.

The writes being executed are all of this type:

INSERT INTO XX ON DUPLICATE KEY UPDATE

It’s frustrating that we can’t roll back after the upgrade…

9
| username: xfworld | Original post link

Before a major version upgrade, it’s better to do a POC, this way you’ll be more confident.

Minor version changes are smaller, mainly focusing on fixes.

There are also those using version 4.X.X who never upgrade because it’s very stable… :rofl:

10
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.