A Large Number of Truncate Table and Update TiFlash Replica Status Tasks

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 大量的truncate table和update tiflash replica status任务

| username: wakaka

【TiDB Usage Environment】 Production Environment
【TiDB Version】 5.0.6
【Encountered Problem】 A large number of DDL tasks suddenly got stuck, including truncate and update tiflash replica status
【Reproduction Path】

【Problem Phenomenon and Impact】
Observed a continuous accumulation of truncate tasks and update tiflash replica status, and only these two types, no alter or similar tasks. At one point, executing admin show ddl jobs showed over 600 tasks.
Not sure why it is getting blocked, theoretically truncate should be quick?

【Attachments】 Relevant logs and monitoring (https://metricstool.pingcap.com/)

If the question is related to performance optimization or fault troubleshooting, please download the script and run it. Please select all and copy-paste the terminal output results for upload.

2
| username: h5n1 | Original post link

admin show ddl jobs list

3
| username: wakaka | Original post link

It looks like there are hundreds of them, as shown below. This happened once a few days ago.

4
| username: Min_Chen | Original post link

DDL is largely stuck.
Please execute ADMIN SHOW DDL; to check which one is the DDL owner.

5
| username: wakaka | Original post link

It’s just a TiDB node, and the CPU, memory, and other monitoring metrics of that node are all within the normal range.

6
| username: wish-PingCAP | Original post link

Hi,

After adding a TiFlash replica, TiDB will continuously initiate an “update tiflash replica status” DDL to update the synchronization progress of the TiFlash replica. This DDL executes very quickly. From the diagram, it appears that since TiDB’s DDL operations are executed sequentially in a queue, it is actually being blocked by the second row DDL “truncate table” (job id = 8988326) in the table. It is recommended to investigate the logs to determine the cause of the “truncate table” DDL blockage.

7
| username: wakaka | Original post link

The truncate statement is also not very clear why it is executing slowly. Normally, it should be fast.

8
| username: alfred | Original post link

If the amount of data being truncated is large, it will also be slow.

9
| username: wakaka | Original post link

Truncate is usually not slow, even with a large amount of data, right?

10
| username: wish-PingCAP | Original post link

I have already replied to this post, please check if it can solve the problem~

11
| username: system | Original post link

This topic will be automatically closed 60 days after the last reply. No new replies are allowed.