What issues might arise if TiCDC is started with an early start_ts?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC 可以通过一个start_ts启动，那这个 start_ts 非常早，会有什么问题吗？

| username: mxd-321

TiCDC can be started with a start_ts. If this start_ts is very early, what issues might arise?
Won’t the binlog of TiKV be purged?

2
| username: h5n1 | Original post link

This start_ts cannot exceed gc_safe_point.

3
| username: neilshen | Original post link

Following @h5n1’s response,

The start_ts cannot exceed gc_safe_point

TiKV does not have a traditional “binlog”. TiCDC obtains historical data by reading historical versions of data stored in TiKV.
When the start ts is earlier than the gc_safe_point, TiCDC cannot read versions earlier than the gc_safe_point, which will cause the synchronization task to fail.

If the start ts is very early, it may lead to:

  • TiCDC may increase the IO pressure on TiKV nodes when reading a large number of historical versions, indirectly affecting business operations.
  • After TiCDC reads the historical versions, it will cache the data on the local disk. If the data volume exceeds the TiCDC disk capacity, it will cause synchronization interruptions.
4
| username: mxd-321 | Original post link

Sure, thank you. TiKV definitely has a change log. May I ask where the change log is stored? I saw on the official website that it is stored in RocksDB?

5
| username: neilshen | Original post link

TiKV does not have a “change log” in the traditional sense. The change data obtained by CDC comes from two sources:

  • The MVCC versions stored in TiKV RocksDB.
  • The Raft log that TiKV writes in real-time.

CDC unifies these two data sources into a complete and continuous change data stream for output.

6
| username: mxd-321 | Original post link

Ingenious, truly ingenious. The MVCC version in RocksDB should be the timestamp, right?

7
| username: neilshen | Original post link

Yes :white_check_mark:

8
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.