TiCDC Large Transaction Synchronization Consumes a Lot of Memory

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc大事务同步消耗大量内存

| username: polars

【TiDB Usage Environment】Production\Test Environment\POC
Production Environment
【TiDB Version】
6.1
【Encountered Problem】
When ticdc synchronizes large transactions, it requires a large amount of memory. If the pre-allocated memory is insufficient, it will cause OOM and continuous restarts, leading to synchronization failure.
As shown in the transaction below, the maximum memory required is 91G. If the reserved memory is insufficient, it will continuously OOM, causing synchronization to fail.

image
image1380×732 38 KB

image
image1380×726 34.5 KB

Please check if resource usage can be optimized.

2
| username: xfworld | Original post link

TiCDC has already described that it does not support large transactions… It is recommended to reasonably split the business logic and transactions.

image
image807×487 30.3 KB

3
| username: CharlesCheung96 | Original post link

CDC has recently optimized support for large transactions. Currently, the OOM issue can only be resolved in version v6.2. If it’s a test environment, it is recommended to update to v6.2 CDC and try it out. You just need to set transaction-atomicity=none in the sink URI.
Additionally, the related optimizations will also be released in the upcoming v6.1.1.

4
| username: polars | Original post link

Great, let’s give it a try.

5
| username: alfred | Original post link

Theoretically, it is difficult for databases to handle large transactions well. Splitting large transactions should be a good choice.

6
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.