TiCDC Cross-Public Network Data Synchronization Solution

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc跨公网同步数据方案

| username: TiDBer_vC2lhR9G

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.0
We need to migrate our database from Alibaba Cloud to Huawei Cloud. There are related solutions in the documentation, but there are some unclear points that I would like to ask everyone about.

Here, I am confused about whether to deploy ticdc on the source TiDB or the target TiDB.
Option 1: Deploy ticdc on the source end, and use --sink-uri= to fill in the public IP address of the target end for synchronization. I have tried this solution, and although the data can be synchronized, the speed is very slow.
Option 2: According to the best practice instructions in the documentation, deploy ticdc on the target end, and use --sink-uri= to fill in the internal IP address of the target end.

However, I don’t quite understand this solution. When synchronizing over the public network, if I deploy ticdc on the target end, how can it pull data from the source end? Moreover, the PDs on both sides maintain their respective internal IP relationships, so the requests cannot go through.

Could the experts please provide some guidance on the solution? Thank you.

| username: hey-hoho | Original post link

The document means that the CDC server is placed in the target network segment, while the CDC instance is still deployed in the source TiDB cluster.

| username: TiDBer_vC2lhR9G | Original post link

Since the target end is in another cloud, if the ticdc instance is managed within the original TiDB cluster, it will obtain the internal IPs of the original PD and TiKV nodes. How will it be able to access them then?

| username: hey-hoho | Original post link

Since you are spanning across clouds, both sides need to have accessible public IPs.

| username: tidb菜鸟一只 | Original post link

If TiCDC is across the public network, and the source end does not have external network access but can access the external network, then it can only be deployed at the source end because deploying at the target end would not allow access to the source end. If the source end can also access the external network, then it should definitely be deployed at the target end. However, for cross-network scenarios, it depends on your bandwidth; the synchronization speed is limited by the bandwidth.

| username: TiDBer_vC2lhR9G | Original post link

Yes, both sides have public IPs and can communicate with each other.

| username: TiDBer_vC2lhR9G | Original post link

Both the source and target ends can access the internet. If TiCDC is deployed in the target data center but the cluster topology belongs to the source end, meaning the PD uses the source end’s PD, then TiCDC will get the internal IPs of the source end’s PD and KV. During migration, it seems that TiCDC in the target data center will use internal IPs to access, which means it won’t be able to pull data from the source end’s TiKV.

tiup ctl:v6.5.0 cdc changefeed create --server=http://xxx.xxx.50.118:8300 --sink-uri="mysql://root:youmai456@192.168.0.101:4000" --changefeed-id="mall-task1" --start-ts="439701707253350615"

Here, I tried using the external IP of the target end’s TiCDC for --server, and --sink-uri to output to the internal IP of the target end’s TiDB cluster. Ultimately, I encountered an error at the --start-ts step, where TiCDC couldn’t get the previous timestamp from the source end for incremental synchronization.

| username: tidb菜鸟一只 | Original post link

Indeed, when deploying TiCDC across public networks, it is unable to obtain changes in the local TiKV at the target end. However, PD can be specified with the public IP and port of PD, and then you can open the IP and port of PD to TiCDC in the security group of Alibaba Cloud…

| username: TiDBer_vC2lhR9G | Original post link

I also tried this operation. I attempted to scale out a public PD for TiCDC, but the original cluster crashed directly, and I even caused a production outage. Check out this post: 救急!!!,生产集群崩了,pd没办法选举 - TiDB 的问答社区.

| username: tidb菜鸟一只 | Original post link

Aren’t your two clusters currently deployed on ECS instances of Alibaba Cloud and Huawei Cloud? Each ECS should be able to open an external IP and port, but you definitely need to deploy using internal IPs because of the low network latency. So now you can only deploy TiCDC on the source end (Alibaba Cloud) and then connect to the target end (Huawei Cloud) for synchronization. If you deploy on the target end (Huawei Cloud), although TiCDC can connect to the external IP of the PD on the source end (Alibaba Cloud), it cannot connect to the internal IP of TiKV, so the second method is indeed not feasible…

| username: TiDBer_vC2lhR9G | Original post link

This solution is OK for transmission, but the CDC output crosses the public network, making the speed very slow, almost like it can’t be synchronized (even though the public network bandwidth is 100M), so it is also unusable.

Do you have any other migration methods?

| username: neilshen | Original post link

For cross-cloud (or cross-region) synchronization, the following operations are recommended:

  • Deploy TiCDC to the downstream, as the network latency from TiCDC to the downstream TiDB is likely to be lower.
  • Ensure the network between upstream and downstream is connected, so that TiCDC can access all TiKV and PD.

At this point, the upper limit of synchronization speed depends on the bandwidth of the network between upstream and downstream.

| username: TiDBer_vC2lhR9G | Original post link

Both the upstream and downstream TiDB clusters are deployed using internal IPs. Could you please advise on how to enable TiCDC to access all TiKV and PD nodes?

| username: tidb狂热爱好者 | Original post link

Use Alibaba Cloud’s DTS to synchronize to Huawei Cloud’s MySQL or TiDB.

| username: TiDBer_vC2lhR9G | Original post link

I noticed that Alibaba Cloud DTS actually supports syncing TiDB. It seems like very few people use this. Have you successfully tried it on your side?

| username: dba-kit | Original post link

This only supports TiDB on the write side, the upstream should still be unsupported.

| username: dba-kit | Original post link

Is there a requirement for the data synchronization cycle? If it’s version 6.5.0, you can actually consider using OSS. By using PiTR, you can synchronize incremental updates to OSS, and Alibaba Cloud can periodically apply the incremental logs. However, this solution is suitable for one-time data switching but not for regular hot backup.

| username: TiDBer_vC2lhR9G | Original post link

I checked it out, and it indirectly supports it. Using Pump and Drainer to output to Kafka, DTS consumes Kafka and then outputs to the target end.

| username: TiDBer_vC2lhR9G | Original post link

We need to continuously perform incremental synchronization, and once the business split is complete and there is no more incremental synchronization, it will be finished. I’ll try this solution to see if it works.

| username: dba-kit | Original post link

PiTR and TiCDC both ensure zero data loss for RPO, but the RTO is somewhat long, requiring periodic application of the changelog, which is sufficient for business verification reads. The only drawback is that both read and write operations need to be switched over together; it’s not possible to switch reads first and then writes.