High Latency in TiCDC Synchronization

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc同步延时高

| username: TiDBer_pkQ5q1l0

[TiDB Usage Environment] Poc
[TiDB Version] v6.5.2
[Encountered Problem: Phenomenon and Impact]
During upstream stress testing, 8 threads are writing, with each thread writing 1 million records, totaling 80 million, all written to a single table. It was found that the downstream application is writing with a single thread, causing the delay to keep increasing. Is there any way to optimize the number of threads writing to the downstream in ticdc?
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
image
image

| username: xfworld | Original post link

There are configuration parameters that can be adjusted when creating a CDC changefeed.

| username: TiDBer_pkQ5q1l0 | Original post link

Okay, I’ll give it a try, thank you.

| username: 像风一样的男子 | Original post link

What is the downstream Kafka database?

| username: TiDBer_pkQ5q1l0 | Original post link

Downstream TIDB

| username: 像风一样的男子 | Original post link

Check the load and IO situation downstream to see if the performance is lagging.

| username: redgame | Original post link

It also looks like IO.

| username: TiDBer_pkQ5q1l0 | Original post link

The IO, CPU, and MEM of each node in the downstream cluster and TICDC are not high. The main issue is that the multi-threading in the upstream becomes single-threaded when executed in the downstream. Although it appears that there are many threads in the downstream, in reality, only one thread is doing the work while the other threads are sleeping. I don’t know why this is happening.

| username: TiDBer_pkQ5q1l0 | Original post link

Additionally, I found another issue. After completion, the downstream had about 800 more records than the upstream. The downstream is set to read-only, so it shouldn’t have more data. Is TiCDC still not mature enough?

Source cluster:

Target cluster:

| username: nongfushanquan | Original post link

Could you provide more detailed information? TiCDC has been relatively stable since version 5.4, and there haven’t been any data inconsistency issues. Also, does the table being synchronized have a PK/UK?

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.