TiCDC Dual Sink Incremental Backup to S3 is Slow

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC双Sink同步增量备份到S3速度慢

| username: Jellybean

[TiDB Usage Environment]
Production / Testing / Poc

[TiDB Version]

  1. TiDB cluster version v5.4.0

  2. TiCDC version is v5.3.0 (v5.4.0 does not support writing to S3, so we reverted to the old version v5.3.0)
    /cdc version
    Release Version: v5.3.0
    Git Commit Hash: 8b816c9717a6d18430f93a6b212f3685c88d4607
    Git Branch: HEAD
    UTC Build Time: 2022-08-10 04:27:19
    Go Version: go version go1.16.13 linux/amd64
    Failpoint Build: false

[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Problem Phenomenon and Impact]

  1. Initially, a master-slave cluster synchronization was deployed in a cluster. After tuning TiCDC, the sink TiDB QPS was optimized from 5k to a maximum of around 45k, with no bottleneck in TiDB cluster master-slave synchronization.
  2. Subsequently, in the same cluster, a new changefeed was started to achieve incremental backup to S3 to meet different backup needs.
    A new TiCDC incremental synchronization task to S3 was created, but the synchronization QPS of this task was only around 4.5k, whereas the master-slave synchronization of the same database table could reach 27k.
    From the task configuration, the upstream tables are exactly the same. The puller, sorter, and mounter of these two tasks are the same, with the difference being in the cdc sink.

The official documentation states that starting from TiDB v6.5.0, TiCDC supports saving row change events to storage services like S3.
Based on practical use, TiCDC in version v5.4.0 does not support writing to S3, so we use the old version v5.3.0 which does support it. Currently, the actual synchronization QPS to S3 is between 4k-5k.

Optimization attempts:

  1. Network optimization confirmed that the write latency between TiCDC nodes and S3 is below 100ms, so high network latency should not be an issue.
  2. Referencing [s3://bucket/prefix?worker-count=128&flush-interval=1m30s&file-size=33554432], adjusting parameters like worker-count did not improve the synchronization speed to S3.
  3. Expanding TiCDC nodes did not significantly improve synchronization speed.

image
image1224×865 131 KB

image
image1241×271 35.4 KB

During operation, the cluster does not have memory, CPU, or IO performance bottlenecks. I would like to know if there are any other ways to improve the efficiency of writing to S3 in this version.

For example, when synchronizing lagging data, a set of comparison data can be seen:
For master-slave synchronization Sink TiDB: puller (xxx:8300-resolved 48k, xxx:8300-kv 25k) → sorter (36k) → mounter (36k) → sink (35k).
For incremental backup Sink S3: puller (xxx:8300-resolved 48k, xxx:8300-kv 25k) → sorter (4.5k) → mounter (4.5k) → sink (4.5k)

The suspected issue is that the sorter is relatively slow. Currently, the per-table-memory-quota is configured to 100MB.

[Resource Configuration]
Resource configuration is as follows:
CPU: amd64, 56 (112 vCore)
Memory: 512GB
Disk: 2.9TB NVME
10 Gigabit Network Card

Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Currently, TiCDC and PD are temporarily mixed. The cluster has 3 TiCDC nodes, with 2 TiKV deployed per machine. During testing and usage, machine resources were sufficient, with no resource bottlenecks.

2
| username: Fly-bird | Original post link

Is there a bottleneck in S3 writes?

3
| username: Jellybean | Original post link

Based on S3, there is local SSD caching, so it is unlikely to be a performance bottleneck when writing to S3.

4
| username: 像风一样的男子 | Original post link

Do you have AWS technical experts for support? Can I consult about the QPS limits of S3 and bucket sharding to improve read and write performance?

5
| username: Jellybean | Original post link

I hadn’t thought of this approach, I’ll consult and take a look.

6
| username: WalterWj | Original post link

After TiCDC writes files, it will optimize parallel file writing. Currently, single tables are written in a single parallel manner.

You can consider writing locally and then using a script to flush to S3 :thinking:, if the performance really can’t be improved.

7
| username: Jellybean | Original post link

So, does that mean version v5.3.0 does not support concurrent Sink S3? Adjusting the worker-count parameter during testing did not take effect.

8
| username: WalterWj | Original post link

Single table operations are single-threaded, while operations between multiple tables are parallel. Does version 5.3.0 of CDC have the file drop feature? This seems to be newly released and not GA yet.

9
| username: Jellybean | Original post link

v5.3.0 supports writing to S3, but it is no longer supported in v5.4.0. There must have been some refactoring done. Due to the cluster version on our side, we can only use v5.3.0.

10
| username: WalterWj | Original post link

Alright, currently you can use BR for PITR. 530 is not an LTS version, right? It’s best to revert to the main version. Also, this feature is not GA yet, so make sure to test it thoroughly.

11
| username: Jellybean | Original post link

v5.3.0 and v5.4.0 are LTS versions and are relatively stable in use.

Recently, I’ve been using CDC scenarios more frequently and encountered the issue of slow writing to S3.

12
| username: Jellybean | Original post link

The latest situation is that by increasing the per-table-memory-quota parameter to 512MB, the peak QPS for writing to S3 can be increased from 5k to 10k, but the QPS is not stable and fluctuates between 5k and 10k.

13
| username: WalterWj | Original post link

The parallelism for single tables in the product might need to wait a bit longer; the current queue for this requirement is not very ideal. It seems like you could try writing to a local NVMe disk first, and then use a script to write to S3 if there are performance requirements.

A bit of delay during peak times is acceptable, so it’s not a big deal.

However, if TiCDC is writing to a file for PITR (Point-in-Time Recovery), it is recommended to use the new version of BR.

14
| username: 像风一样的男子 | Original post link

I used binlog to file for PITR, and I’ve practiced recovery a few times without any issues.

15
| username: Jellybean | Original post link

The new version of BR PITR can indeed meet our needs. However, the online cluster version is v5.4, and it is difficult to upgrade in a short time.

Are you referring to CDC writing to the local disk, is it achieved by CDC writing to NFS?

16
| username: Jellybean | Original post link

Based on our previous testing and usage of tidb-server’s binlog, considering the overall stability of the tool and future official support, we mainly recommend the CDC tool.

17
| username: WalterWj | Original post link

Yes. Writing memory to disk is relatively fast…

18
| username: Jellybean | Original post link

Understood, thank you for your patient explanation.
We previously compared using NFS and S3, and later it was uniformly required to use S3. For now, we are temporarily increasing the per-table-memory-quota parameter to improve synchronization performance. It seems that to fundamentally solve the performance issue of writing to S3, we still need to upgrade the cluster to a higher version and then use the BR’s PITR feature or TiCDC after v6.5.

19
| username: Jellybean | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.