Data Migration (MySQL to TiDB): MySQL has 100 tables, and the pre-check for the migration task takes a long time. How to optimize?

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 数据迁移(mysql->tidb),mysql有100张表,迁移任务前置检查花费很长时间,如何优化?

| username: TiDBer_b1iRkG7I

I need to synchronize MySQL data to TiDB. The MySQL database has more than 100 tables. The pre-check of the migration task is stuck and unresponsive. I suspect it is because there are too many tables. How can I optimize this?

| username: 小龙虾爱大龙虾 | Original post link

Several people in the community have asked about this dmctl freezing issue, and it seems there aren’t many logs. Do you have any logs for this, from the dm worker node and dm master node?

| username: WalterWj | Original post link

Do you have a lot of partitioned tables?

| username: dba远航 | Original post link

First, check the configuration file and network status.

| username: redgame | Original post link

You can reduce it by half and see if it’s possible to find out which tables have issues. Sometimes it’s just a problem with one table.

| username: TiDBer_5cwU0ltE | Original post link

When encountering such a hang, if the logs do not provide any hints, you should check the system call situation. To see where it is stuck, I recommend a tool: strace.
Reference command:

strace -f -F -o ~/straceout.txt $COMMAND
| username: 小于同学 | Original post link

Check the network status.

| username: TiDBer_小阿飞 | Original post link

The link you provided is a great resource for understanding the latest features and improvements in TiDB. It offers detailed insights into the architecture, performance enhancements, and new functionalities that can help optimize your database management and operations.

| username: TiDBer_小阿飞 | Original post link

Were these two posts made by the same person?

| username: residentevil | Original post link

If only migrating full data without considering incremental data, can we use mysqldump to export from MySQL and then import into TiDB, with 10 tables per task and 10 concurrent tasks? This way, the efficiency would be higher.

| username: TiDBer_b1iRkG7I | Original post link

Yes, the same issue.

| username: TiDBer_b1iRkG7I | Original post link

Without logs, on my side, it’s caused by having too many upstream and downstream tables. If only one table is synchronized, the check will be very fast.

| username: 哈喽沃德 | Original post link

It can be done in batches.

| username: TiDBer_aaO4sU46 | Original post link

Try reducing the points.

| username: TiDBer_RjzUpGDL | Original post link

Try excluding the large table first.

| username: zhang_2023 | Original post link

The probability of a network issue is higher.

| username: kelvin | Original post link

Try several tables at once.