TiCDC Performance Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc 性能问题

| username: TiDBer_20QjYTLl

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] v6.5.0
[Reproduction Path]
Using ticdc for incremental data synchronization between different clusters
[Encountered Problem: Problem Phenomenon and Impact]
Data synchronization between two clusters is relatively slow, with the checkpoint lag reaching up to 1 hour at its peak.
After checking the monitoring, the puller’s TPS is at 200,000/s, but it immediately drops to 20,000 at the sorter stage. Could you please explain what might be causing this situation?

| username: 像风一样的男子 | Original post link

How is the disk performance of CDC? It isn’t a mechanical disk, right?

| username: TiDBer_20QjYTLl | Original post link

It’s a mechanical hard drive.

| username: 像风一样的男子 | Original post link

If the server uses mechanical hard drives or other storage devices with latency or throughput bottlenecks, the performance of Unified Sorter will be significantly affected.

| username: TiDBer_20QjYTLl | Original post link

Can we switch to a different sorter?

| username: 像风一样的男子 | Original post link

Your data volume seems quite large. You can try adding CDC nodes and splitting tasks to distribute the load across different servers.

| username: TiDBer_20QjYTLl | Original post link

  1. I’m using version 6.5.0. Why is the default sorter still unified?
  2. How do I split tasks? Are there any parameters that can be specified?
| username: MrSylar | Original post link

Try increasing this value?

[sorter]
The Sorter module sets the size of the pebble block cache shared by the 8 default-started pebble DBs, in MiB, with a default value of 128.
cache-size-in-mb = 128

| username: 像风一样的男子 | Original post link

You are definitely synchronizing more than one table. Just split the synchronization tasks by table.

| username: TiDBer_20QjYTLl | Original post link

I am now just a table.

| username: TiDBer_20QjYTLl | Original post link

The key second question is why my version 6.5.0 is still using uni sorter.

| username: 这里介绍不了我 | Original post link

Does the synchronized table have partitions?

| username: 这里介绍不了我 | Original post link

I think this article is well-written and worth a read.
Column - 10x Improvement - TiCDC Performance Tuning Practice | TiDB Community

| username: TiDBer_20QjYTLl | Original post link

I have seen this. My scenario involves only one table, and it is not a table with a single row exceeding 1k.

| username: WalterWj | Original post link

I recommend upgrading to the latest version 6.5, for example, try 659.