Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: 使用dm工具,同步汇集mysql库中每天建新表的情况出错
[Test Environment for TiDB]
[TiDB Version]
Release Version: v6.5.1
Edition: Community
Git Commit Hash: 4084b077d615f9dc0a41cf2e30bc6e1a02332df2
Git Branch: heads/refs/tags/v6.5.1
UTC Build Time: 2023-03-07 16:12:08
GoVersion: go1.19.5
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv
[DM Version]
v6.6.0
[Reproduction Path]
The upstream log table adds a new table every day. For example, today a new table log_20230427 is created to store data, and tomorrow it will become log_20230428 to store data.
Because I want to aggregate multiple upstream data into one TiDB table,
the DM route configuration is as follows:
route-Log:
schema-pattern: “China_Log”
table-pattern: “Log_"
target-schema: downStream
target-table: “Log_date”
extract-table:
table-regexp: "Log_(.)”
target-column: “c_table”
extract-schema:
schema-regexp: “(.)"
target-column: “c_schema”
extract-source:
source-regexp: "(.)”
target-column: “c_source”
route-log_all:
schema-pattern: “China_Log”
target-schema: downStream
[Encountered Problem: Phenomenon and Impact]
The current problem is that there is no issue with synchronizing the data of the day and the stock data. However, after a day passes and the upstream generates a new table, DM synchronization will report an error similar to the following:
“Message”: “startLocation: [position: (mysql-bin|000001.000001, 4540655), gtid-set: 00000000-0000-0000-0000-000000000000:0], endLocation: [position: (mysql-bin|000001.000001, 4540774), gtid-set: 00000000-0000-0000-0000-000000000000:0]: gen insert sqls failed, sourceTable: China_Log
.Log_2023_04_27
, targetTable: downStream
.Log_date
: Column count doesn’t match value count: 20 (columns) vs 23 (values)”,
indicating that the new table has three fewer fields than the downstream summary table, and synchronization cannot continue.
I want to know how to solve this situation?