TiDB TiCDC Synchronization Performance Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIDB ticdc同步性能问题

| username: residentevil

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.8
[Encountered Issue: Issue Phenomenon and Impact] During the process of ticdc synchronizing data from TIDB to TIDB cluster, excluding DDL operation scenarios, has anyone encountered synchronization performance issues? For example, when QPS write && row changes per second reach XX, the ticdc replication delay starts to increase.

| username: zhanggame1 | Original post link

If you only look at DML statements, there will be a delay if updating a large table; otherwise, it is not noticeable.

| username: residentevil | Original post link

TiCDC achieves data synchronization by parsing the underlying WAL (or Raft logs). In theory, does it have nothing to do with QPS and SQL writing?

| username: 江湖故人 | Original post link

CDC pulls the row change information from the source, so when the number of rows changes consistently, the effect is the same whether it is achieved with one SQL or two SQLs.

| username: residentevil | Original post link

It seems that if a single SQL update affects a large number of rows [for example: update tab set where col0=‘a’ and col1=‘b’, using an auxiliary index for the filter condition, retrieving tens of thousands of rows], the latency might indeed increase.

| username: 江湖故人 | Original post link

Yes, because it is row-based.

| username: 路在何chu | Original post link

What I have seen are all large transaction delays.

| username: residentevil | Original post link

Have you used TiCDC in your business? Are there many issues, haha?

| username: 江湖故人 | Original post link

Large transactions involve more operations, so higher latency is normal. :grinning:
Starting from version v6.2, TiCDC supports the feature of splitting single-table transactions, which can significantly reduce the latency and memory consumption of synchronizing large transactions. Therefore, in scenarios where the atomicity of transactions is not highly required, it is recommended to enable the transaction splitting feature by setting the sink URI parameter transaction-atomicity to address potential synchronization delays and OOM issues.

| username: 江湖故人 | Original post link

The image you provided cannot be translated directly as it is a visual format. Please provide the text content for translation.

| username: residentevil | Original post link

Is the OOM issue referring to the TICDC instance? Speaking of which, there’s another TICDC deployment question. According to the community documentation, it is recommended to deploy TICDC on the TiDB cluster [considering synchronization delay issues]. This seems to have strong coupling. If synchronization delay is not a concern, wouldn’t it be more user-friendly to deploy it separately?

| username: FutureDB | Original post link

TiCDC is a component that you can deploy on a separate machine and add to the TiDB cluster.

| username: 江湖故人 | Original post link

The OOM here could be caused by TiCDC itself or the downstream TiDB.
TiCDC is a component of TiDB and cannot exist independently. You can deploy it together when deploying TiDB, or scale out this component on an existing TiDB. Deploying it separately is definitely easier to maintain than mixed deployment.

| username: redgame | Original post link

Do adjusting the parameters cdc.region-concurrency and cdc.sink-parallelism have any effect?

| username: residentevil | Original post link

Still in the solution research phase, haha.

| username: residentevil | Original post link

I’ll try to deploy it and test it out. Thanks, expert!