Anomalous Phenomenon Discovered in TiDB Table Merging

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 合表发现一个异常现象

| username: Hacker_suny

[TiDB Usage Environment] Production Environment
[TiDB Version] 4.0.0
[DM Version] Cluster version: v2.0.0
[Reproduction Path] Configured a task for sharding merge
There is a business database db with sharded tables t_order upstream, and downstream TiDB needs to merge the sharded tables t_order into t_order.

The main task configuration is as follows:
name: sharding_merge # Task name, must be globally unique
task-mode: all # Task mode, can be set to “full”, “incremental”, “all”
ignore-checking-items: [“auto_increment_ID”,“table_schema”]
shard-mode: “pessimistic”

routes:
db-route-rule:
schema-pattern: “db”
target-schema: “sharding_db”
order-route-rule:
schema-pattern: “db”
table-pattern: “t_order_[0-63]”
target-schema: “sharding_db”
target-table: “t_order”

Since there is a t_order table upstream, the dm synchronization ignored the upstream t_order table
black-white-list:
bw-rule-1:
do-dbs: [“db”]
ignore-tables:
- db-name: “db”
tbl-name: “t_order”

[Encountered Problem: Phenomenon and Impact]
Encountered a very strange phenomenon
I have 64 sharded tables in the upstream db, but the downstream TiDB lacks tables 0-6, only having tables 7-63

The strange phenomenon is
When I query the merged t_order table, I find that the full and incremental data of tables 0-6 exist in the t_order table, and there is no related information about tables 0-6 in the dm-work logs.

I would like to ask the experts, has anyone encountered this situation? What is the cause of this phenomenon?

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: Meditator | Original post link

  1. This sounds too fantastical, we need the expert with the wings of fantasy to solve it.
  2. Are you sure there are no other DM tasks? Or issues with DM meta metadata?
| username: Hacker_suny | Original post link

  1. There are other DM tasks, but the task names are unique, so different tasks should not affect each other.
  2. The metadata of dm_meta has been checked and there are no issues. The missing table data can be seen imported in the loader table.
| username: liuis | Original post link

Try re-importing it?

| username: Hacker_suny | Original post link

I have this plan, I’ll try again on the weekend.

| username: liuis | Original post link

Well, try again to see if the problem still exists.

| username: dba-kit | Original post link

It should be a problem with the regular expression. Try “~^t_order_[0-9]+$”, and see if it works correctly.

| username: dba-kit | Original post link

However, theoretically, if you didn’t add a tilde (~) at the beginning and used TiDB’s unique matching rules, which only support * and ?, your approach would be invalid. All tables should not match, rather than just missing tables 0-6, which is quite strange.

| username: liuis | Original post link

Is it okay to rerun?

| username: tidb菜鸟一只 | Original post link

It should be a problem with the regular expression.

| username: Jellybean | Original post link

It looks like there is an issue with the regular expression.

| username: 考试没答案 | Original post link

Just a quick question: When exporting the full data, did you check if there are any numbers from 0-6? Also, has the data been exported? Please check everything. Source end — temporary file: full data, incremental data — into the target database. First, check if the data has been exported from the source end.

| username: Hacker_suny | Original post link

If there is an issue with the regular expression, it will not pass during the check-task.

| username: Hacker_suny | Original post link

When checking, files numbered 0-6 do exist.

| username: Hacker_suny | Original post link

Just started rerunning, there were issues with the machine over the weekend.

| username: dba-kit | Original post link

There is an issue with the regex, it will only result in a mismatch and will not cause an error.

| username: Hacker_suny | Original post link

But if there is a problem, it should all be lost.

| username: Hacker_suny | Original post link

Moreover, it’s very strange that the data actually came through, but the table structure did not.

| username: liuis | Original post link

What does it mean when the data comes over but the table structure does not?

| username: dba-kit | Original post link

Then I probably guessed it. Normally, after using shard-mode, the downstream should only have the merged t_order table, not the sharded t_order_xx tables. The regex t_order_[0-63] you wrote only matched tables [0-6], so only the data from the sharded tables [0-6] was written into t_order. The other sharded tables [7-63] were treated as regular single tables and synchronized separately.