I would like to consult about the migration plan for approximately 10TB of data

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 想咨询一下10T左右数据的迁移方案

| username: TiDBer_WlslcXgH

I would like to inquire about the service deployment plan for data migration. We need to synchronize around 10TB of data from 31 MySQL databases to TiDB. How should we deploy and configure it to make it faster?

| username: tidb菜鸟一只 | Original post link

You need to use DM for this.

| username: xingzhenxiang | Original post link

dumpling for export
lighting for import
dm for real-time synchronization

| username: yiduoyunQ | Original post link

| username: Jellybean | Original post link

This is a classic question about migrating from MySQL to TiDB. The general approach is as follows:

  • Use Dumpling + TiDB Lightning physical import mode for full data migration.
  • After completing the full data migration, use DM to complete the incremental data migration.

Afterwards, the application can switch to TiDB, either by stopping the service for the switch or by implementing dual-write for a smooth migration.

| username: Fly-bird | Original post link

First full data migration: Use Dumpling + TiDB Lightning
Then incremental data migration: Use DM to complete incremental data migration

| username: 像风一样的男子 | Original post link

The official documentation has detailed tutorials:

| username: 舞动梦灵 | Original post link

You can directly use mysqldump to export and import into TiDB, then use DM to start real-time synchronization from the export time point. This is how I did it, but the data volume wasn’t that large. One thing to note is to check if the user table in MySQL is the same on both sides. I have encountered situations where the user table was different.

| username: TiDBer_小阿飞 | Original post link

Merge 31 databases into one in TiDB, or migrate them one by one? Full dump with Dumpling, incremental with CDC.

| username: 路在何chu | Original post link

Migrate them one by one, create 11 tasks, and just filter and synchronize.

| username: 舞动梦灵 | Original post link

Do it one by one. Do not repeat the database names.

| username: Jolyne | Original post link

We previously had a scenario with around 7TB of data, using Dumpling for export and Lightning for import. If we want to maintain data consistency directly in the cluster afterward, we use DM. However, it’s best to export and import in batches when using Dumpling and Lightning.

| username: TiDBer_QYr0vohO | Original post link

  1. Use Dumpling and Lightning for full data export and import.
  2. Use DM for subsequent incremental data migration.
| username: 随缘天空 | Original post link

It is more appropriate to use BR for large data volumes and high performance. You can refer to the following link: https://docs.pingcap.com/zh/tidb/stable/migrate-from-tidb-to-tidb

| username: 像风一样的男子 | Original post link

How to use BR with upstream MySQL?

| username: 随缘天空 | Original post link

Refer to the following article: Migrating Data from a Large MySQL Database to TiDB

| username: 像风一样的男子 | Original post link

Are you a robot? You reply without even looking at the content.

| username: Kongdom | Original post link

Indeed, it seems like a robot. The linked document doesn’t mention BR at all.

| username: 随缘天空 | Original post link

I want to ask how you manage to reply without looking at the content. The question is about large data migration solutions. Do I really have to read all the replies before I have the right to answer?

| username: 随缘天空 | Original post link

Are you answering questions without thinking? Please check if BR can be used for large data migration before responding. It’s just that the source is not MySQL, but it was corrected later, right?