DM Synchronization Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dm同步问题

| username: mono

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.7
[Encountered Issue: Problem Description and Impact]
Currently, we are using DM to synchronize MySQL data to the TiDB cluster. We now have a requirement involving two data sources, each with a MySQL table t1. However, the fields in the two tables are not completely identical. I would like to ask if there is a way to merge and synchronize some fields from these two MySQL tables into a single table in TiDB?

| username: tidb菜鸟一只 | Original post link

Just take the union of the columns from the two upstream tables for the downstream table, and then configure the routing.

| username: mono | Original post link

It’s not a simple table merge. There are two databases, each with one table. The table t1 in database a has fields (a, b, c, d), and the table t1 in database b has fields (a, d, e). We need to merge these two tables and synchronize them to the table t1(a, b, d, e) in TiDB. If possible, could you provide a configuration for reference? Thank you!

| username: forever | Original post link

DM alone probably can’t achieve this; it doesn’t support such complex configurations. You can synchronize to the target end and then process it, or the target can directly use views.

| username: onlyacat | Original post link

Currently, DM only supports table-level synchronization. For field-level synchronization, you might need to look for other tools or submit a request to the official team.

| username: 小于同学 | Original post link

DM is currently not supported.

| username: mono | Original post link

Thank you, everyone. Got it! :handshake:

| username: GreenGuan | Original post link

DM cannot achieve this. You can try merging the two tables into one on the MySQL side (e.g., using a trigger) and then synchronize through DM.

| username: TiDBer_小阿飞 | Original post link

Migration with Different Number of Columns Upstream and Downstream

| username: mono | Original post link

It’s not the same scenario. It is not supported.

| username: jonathon | Original post link

Please provide the detailed version information.

| username: Jolyne | Original post link

I’m simply curious about the scenario where you need both the a and d fields from t1 and the a and d fields from t2. If it’s not directly inserting into a union table, do the two tables have duplicates?

| username: TiDBer_rvITcue9 | Original post link

Why not try creating a couple of tables downstream?

| username: mono | Original post link

The table has many columns, and it has been vertically partitioned in MySQL.

| username: tidb菜鸟一只 | Original post link

I tested it, and it seems that it really doesn’t work. Synchronization will check the structure of the tables.

| username: TiDBer_LM | Original post link

If you succeed, remember to write an article so we can all learn from it.

| username: mono | Original post link

It didn’t work. It doesn’t support it! :joy:

| username: kelvin | Original post link

I feel that DM currently doesn’t support it. The minimum level of support is table-level synchronization.

| username: TiDBer_HErMeXDz | Original post link

This kind of requirement is a bit abnormal… Even if it’s merged, the business still needs to adapt.