[Q&A Review] TiCDC Source Code Analysis #7: Unveiling the TiCDC Sorter Module

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 【Q&A 回顾】TiCDC 源码解读#7 TiCDC Sorter 模块揭秘

| username: TiDB社区小助手

This article is a summary of the Q&A and video review for the seventh session of TiCDC Source Code Interpretation - TiCDC Sorter Module Revealed, along with a collection of downloadable sharing materials. If you have more to discuss about this session’s content, feel free to leave a comment below.

Q&A Review of This Session

Below is the Q&A review for this session “TiCDC Sorter Module Revealed”:

Q1: Can you share more details about the large transaction splitting? Is the 64KB requirement for cross-table or the same table?
A: This session mentioned that the key encoding of events uses CommitTs and StartTs as higher prefixes than Record, so when scanning events from the SortEngine, they are sorted by CommitTs and StartTs, thus events from the same transaction are aggregated together. When the large transaction splitting feature is enabled, for each task generated on a table, the transaction is split every 64KB, and part of the events are written to the downstream. However, after splitting the large transaction, it still needs to wait for the entire transaction to be written to the downstream before advancing the Checkpoint.

Q2: Which version can limit the memory size of cdc or sorter?
A: Versions 5.4 and later can limit the memory usage of the sorting engine itself through max-memory-percentage. The memory usage of changefeed can be limited through the memory-quota item in the changefeed configuration in versions 6.5 and later.

Q3: Which parameter controls the memory of ticdc or sorter? Currently, only per-table-memory-quota is found.
A: In versions 6.4 and earlier, per-table-memory-quota limits the amount of data waiting to be sent to the downstream on each table, with a default of 10MB. This option was replaced by memory-quota in version 6.5, which can be set on each changefeed. The parameter that limits the total memory usage of the sorting engine within the TiCDC process is max-memory-percentage.

Announcement of Live Interaction Winners

Congratulations to the following users who participated in the interaction and won prizes! You can receive 100 TiDB community points/question.

The following users can add WeChat ID: Oneandtwii before March 10, 2023, and reply with: Your Video Account Nickname + TiDB Community Nickname to redeem points.

No. Video Account User
1 俨-zen
2 xz
3 冯老爷

If you have more to discuss about this session’s content, feel free to leave a comment below.