Choosing Synchronization Tools

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 同步工具的选择

| username: shuyu_zhihui

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 7.4.0
The TiDB synchronization tools I know of include DM, TiCDC, Dump, Binlog, Drainer, T2O, and Lightning. Besides these, there might be other synchronization tools. Some are for synchronizing to Oracle, some to MySQL, some can do full synchronization, and some can do partial synchronization.

  1. Why are there so many synchronization tools?
  2. What are the differences between them?
  3. Are there any redundant tools?
  4. How should one choose when doing synchronization?

The practical introductions in the documentation are not very clear. Are there any related articles or materials you can recommend?
Thank you, experts, for your help.

| username: ffeenn | Original post link

In your case, you should use ETL tools.

| username: 芮芮是产品 | Original post link

In fact, for synchronization, there’s only CloudCanal.
For syncing from MySQL to TiDB, use DM.
For syncing from TiDB to TiDB, use TiCDC.
Everything else is history.

| username: TiDBer_小阿飞 | Original post link

dm, ticdc, dump, binlog, dranier, t2o, lighting

| username: 小龙虾爱大龙虾 | Original post link

What is the t2o tool? Can someone provide a link?

| username: Billmay表妹 | Original post link

  1. Why are there so many synchronization tools?

The existence of so many synchronization tools is to meet different business needs and scenarios. Factors such as different database systems, data scales, and data synchronization methods all lead to the need for different synchronization tools to perform data synchronization. For example, TiDB, as a distributed database system, can synchronize data with other database systems (such as Oracle, MySQL), thus requiring different tools to achieve data synchronization with different database systems.

  1. What are the differences between them?
  • DM (Data Migration): DM is an open-source, easy-to-use database migration tool used to migrate data from MySQL/MariaDB to TiDB. It supports full migration and incremental synchronization, providing data consistency guarantees and high availability.

  • TiCDC (TiDB Change Data Capture): TiCDC is an open-source, distributed data change capture tool used to synchronize data changes in a TiDB cluster to other data storage systems. It supports streaming data changes to other systems, such as Kafka, MySQL, etc.

  • Dump & Binlog: Dump and Binlog are built-in MySQL tools. Dump is used to export the full data of a MySQL database, while Binlog records incremental changes in the MySQL database. These two tools can be used in conjunction with TiDB to achieve data migration and incremental synchronization from MySQL to TiDB.

  • Drainer: Drainer is a tool used to synchronize MySQL Binlog data to TiDB. It can parse and synchronize MySQL Binlog data to a TiDB cluster, achieving data synchronization from MySQL to TiDB.

  • T2O (TiDB to Oracle): T2O is a tool used to synchronize TiDB data to an Oracle database. It can export and synchronize data from TiDB to an Oracle database, achieving data synchronization from TiDB to Oracle.

  • Lightning: Lightning is a tool used to import large-scale data into TiDB. It can efficiently import data from external storage (such as MySQL, HDFS, etc.) into TiDB, used for quickly initializing or reloading a TiDB cluster.

These tools differ in functionality and usage scenarios, and the appropriate tool can be chosen based on specific needs.

  1. Are there redundant tools?

There may be some overlap and redundancy in certain aspects. For example, both DM and TiCDC can achieve data synchronization between TiDB and other database systems, but their design goals and usage methods are different. DM is more suitable for data migration and synchronization from MySQL to TiDB, while TiCDC is more suitable for synchronizing TiDB data changes to other systems.

  1. How should one choose when doing synchronization?

When choosing a synchronization tool, it should be evaluated and selected based on specific business needs and scenarios. Here are some considerations:

  • Data source and target database: Determine the data source and target database that need to be synchronized, and their types (such as MySQL, Oracle, etc.).

  • Data scale and performance requirements: Evaluate the data scale and synchronization performance requirements, and choose a tool that can meet the needs.

  • Synchronization method: Determine whether full synchronization or incremental synchronization is needed, and whether real-time synchronization is required.

  • Maturity and stability of the tool: Consider factors such as the activity of the developer community, completeness of documentation, stability, and reliability of the tool.

  • Deployment and maintenance costs: Evaluate the deployment and maintenance costs of the tool, including configuration complexity, learning curve, and operational workload.

By comprehensively considering the above factors, choose the synchronization tool that suits your business needs and scenarios.

| username: 春风十里 | Original post link

Dumpling and Lightning are logical backup and restore tools, and cannot be considered data synchronization tools.

| username: shuyu_zhihui | Original post link

t2o drainer, synchronized to the Oracle database.

| username: dba远航 | Original post link

Go watch training video 303, it has everything you want.

| username: come_true | Original post link

There are definitely many synchronization tools to meet different needs. Each synchronization tool has its own characteristics and advantages, so choose according to your needs.

| username: come_true | Original post link

When performing synchronization, you should consider factors such as the size of the backup, whether you can shut down the system, whether it is physical or logical, hot backup or warm backup, and whether you can lock the tables.

| username: 小龙虾爱大龙虾 | Original post link

:+1: Learned something new.

| username: 像风一样的男子 | Original post link

To meet the synchronization needs between different databases.

| username: oceanzhang | Original post link

Drainer is a component of binlog, while dump and lightning belong to the logical backup and recovery components and are not considered synchronization tools.

| username: forever | Original post link

It can also be considered a full synchronization tool.

| username: Kongdom | Original post link

Take a look at the documentation for this use case.

| username: kelvin | Original post link

Here to learn, leaving a comment.