TiCDC Synchronization Data Log Exception from Specified TS

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc从指定TS同步数据日志异常

| username: jaybing926

[TiDB Usage Environment] Production Environment
[TiDB Version] v4.0.9
[Reproduction Path] Operations performed that led to the issue
The old TiDB cluster has weak performance (ordinary SATA disks, high IO, and other services running outside the TiDB components, causing significant pressure). A new cluster with the same version v4.0.9 has been deployed, and data is being synchronized.

  1. First, use br restore to restore the full data to the new cluster.
  2. Then use ticdc to append and synchronize data based on the TS of the full data.

[Encountered Issue: Phenomenon and Impact]
There are many abnormal warning logs in the ticdc logs. I would like to ask what this means and whether it has any impact?

PS:
Our data volume: The full data is approximately 4T * 3 replicas = around 12T.
We perform a full data backup on the 1st of every month. From the 1st to the 13th, the additional data volume is about 100G. Does this large additional data volume have any impact?

[Attachments: Screenshots/Logs/Monitoring]

Looking at the task status, the state has always been normal, but the checkpoint has not changed. Is it waiting for the additional data to be exported? Does it need to calculate the additional data before starting to write to the new database? Is this the logic?

| username: tidb狂热爱好者 | Original post link

Is the performance not enough? CDC also needs to borrow more.

| username: jaybing926 | Original post link

Two CDC nodes, both are new servers with 256GB and 48 cores.

| username: jaybing926 | Original post link

There are many of these Warn logs, I don’t understand what they mean, but the task status is normal.
Now the data directory of ticdc is 130G, why is it so large?

| username: jaybing926 | Original post link

The performance of the new server is sufficient, but the performance of the old cluster is quite low and under a lot of pressure. Could it be a performance issue with the old cluster? Can we confirm that these logs are normal and can be ignored?

| username: CuteRay | Original post link

This is normal. When CDC synchronizes data, it pulls the incremental data locally for processing before synchronizing it downstream.
TiCDC Architecture and Principles | PingCAP Documentation Center

| username: CuteRay | Original post link

Generally speaking, if there is no ERROR exception, you can leave it for now and observe for a while before taking any action.

| username: jaybing926 | Original post link

Okay, I’ll take a look, thanks for the response~

I have another question: TICDC does not support database-level synchronization, right? I couldn’t find the relevant parameters.

Our backup is based on the database, with different backup times for different databases, and different recovery times as well. But when I use ticdc, I specify the TSO of the earliest database backup. Will this cause data duplication or any other issues for the other databases?

| username: asddongmen | Original post link

  1. Basic library synchronization is supported, and you can use filter rule to specify the libraries that need to be synchronized by changefeed. It is recommended that the number of tables synchronized by each synchronization task should not exceed 1000.
  2. The synchronization progress is stuck because there is 100GB of stock data that needs to be pulled over first before synchronization can proceed.
  3. The files in the TiCDC directory have exceeded 130GB because the pulled data needs to be sorted.
  4. Does the large amount of incremental data have an impact: Yes, a large amount of incremental data may cause the synchronization task to be stuck for a long time after initialization is complete.
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.