Just Ask: Migrating TiCDC Message Subscription to Another Kafka Cluster After Migrating TiDB Cluster

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 不懂就问:迁移TIDB集群后,迁移 ticdc消息订阅到其它kafka集群

| username: TiDBer_yyy

[TiDB Usage Environment] Production Environment
[TiDB Version] 5.0.4
Cluster Topology

The TSO of upstream and downstream TiDB is inconsistent, so the checkpoint of DC1’s ticdc cannot be used as the start-ts of DC4’s ticdc, and double writing to downstream Kafka is required.

Questions:

  1. Is es unique at the TiDB cluster level or at the physical (all clusters use this one timestamp) level?

  2. With a network delay of 3-7ms between the two DCs, can es be used by downstream consumers as the unique identifier for APP consumption of DC4 Kafka?
    (Allowing for a certain degree of duplicate consumption.)

| username: tidb狂热爱好者 | Original post link

I don’t understand, so I can’t help you.

| username: TiDBer_yyy | Original post link

Still, thank you, master.

| username: wzf0072 | Original post link

As long as the number of clusters is large enough, transactions are frequent enough, or the time is long enough, duplication will occur in ES.

My understanding is that it is unique within the cluster.

| username: TiDBer_yyy | Original post link

I have already thought of a solution. Thank you, everyone.
Using:

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.