Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: 【Q&A 回顾】TiCDC 源码解读#1 | TiCDC 整体架构概览
|
username: YY-ha
This article is the first issue of TiCDC source code interpretation - an overview of TiCDC’s overall architecture, including a Q&A session and a collection of video reviews and downloadable sharing materials. If you have more questions about the content of this issue, feel free to leave a comment below.
- Video replay: TiCDC 源码解读 | TiCDC 整体架构概览_哔哩哔哩_bilibili
- Download sharing materials: Introduction to TiCDC-2.pdf (1.9 MB)
- Article summary: 专栏 - TiCDC 源码解读（1）-- TiCDC 的架构概览 | TiDB 社区
- Detailed review of the entire TiCDC source code interpretation series: 【Resource Summary】The Most Comprehensive Resources of TiCDC Source Code Interpretation Series!!!
Below is the Q&A review of this issue’s overview of TiCDC’s overall architecture:
Q: TiCDC used to cause OOM due to large transactions. Has this problem been solved?
A: After version 6.1.1, you can solve the large transaction problem by enabling the split transaction switch.
Q: TiCDC currently synchronizes by table. If a single table becomes a hotspot, it cannot utilize the distributed machine resources of CDC. Will this problem be solved in the future?
A: This issue is currently under development.
Q: Can it support capturing full data simultaneously?
A: It depends on the size of the full data. If the full data exceeds 400G, it is recommended to use BR for recovery first.
Q: CDC pulls incremental data without errors, but the logs report region not found. Checking the upstream region shows some regions not found, but no anomalies are found in PD. How to handle this?
A: If there are no errors, no action is needed. CDC has corresponding fault tolerance mechanisms.
Q: The current production cluster has 800 changefeeds, and the synchronization speed is very slow. Is there any way to optimize this? Testing shows that the fewer the changefeeds, the faster the synchronization.
A: In what scenario would you need 800 changefeeds? One changefeed can synchronize multiple tables. It is recommended to keep the number of changefeeds under 100.
Q: Can the Craft protocol be used currently?
A: It is not GA, so it is not recommended to use it.
Q: Separating incremental and full synchronization is too troublesome. Will the full synchronization idea from this year’s Hackathon be considered for implementation?
A: It will be considered, but there are no development plans at the moment.
|
username: