How to Configure Two DM Configuration Rule Names

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: DM配置规则名称怎么配置两个

| username: Ming

[Test Environment for TiDB] Testing
[TiDB Version] v6.5.6
[Encountered Issue: Problem Description and Impact]
May I ask, in the DM task file, how to configure the rule name to be two?

| username: tidb菜鸟一只 | Original post link

Here it is:

block-allow-list:             # If the DM version is earlier than v2.0.0-beta.2, use black-white-list.
  rule-1:
    do-dbs: ["test*"]         # Not starting with the ~ character indicates that the rule is a wildcard; wildcard rules are supported from v1.0.5 onwards.
    do-tables:
    - db-name: "test[123]"    # Matches test1, test2, test3.
      tbl-name: "t[1-5]"      # Matches t1, t2, t3, t4, t5.
    - db-name: "test"
      tbl-name: "t"
  rule-2:
    do-dbs: ["~^test.*"]      # Starting with the ~ character indicates that the rule is a regular expression.
    ignore-dbs: ["mysql"]
    do-tables:
    - db-name: "~^test.*"
      tbl-name: "~^t.*"
    - db-name: "test"
      tbl-name: "t"
    ignore-tables:
    - db-name: "test"
      tbl-name: "log"
| username: Ming | Original post link

One is in the function and the other is in the instance. I want to know how to define two filter rule names in the instance.

| username: Ming | Original post link

I have another question I’d like to ask. Will the rule-2 section of this DM configuration file take effect? I remember that if do-tables and ignore-tables are in the same section, only one of them will take effect.

| username: tidb菜鸟一只 | Original post link

You can define multiple rules within one rule, for example like this:

block-allow-list:                    # Define the filtering rules for migrating tables from the data source, multiple rules can be defined. If the DM version is earlier than v2.0.0-beta.2, use black-white-list
  bw-rule-1:                         # Rule name
    do-dbs: ["~^test.*", "user"]     # Databases to migrate
    ignore-dbs: ["mysql", "account"] # Databases to ignore
    do-tables:                       # Tables to migrate
    - db-name: "~^test.*"
      tbl-name: "~^t.*"
    - db-name: "user"
      tbl-name: "information"
  bw-rule-2:                         # Rule name
    ignore-tables:                   # Tables to ignore
    - db-name: "user"
      tbl-name: "log"

Generally, this configuration with multiple rules is aimed at different data sources. For the same data source, it is impossible to have a rule that matches a table to not sync and another rule that matches the same table to sync…

| username: Ming | Original post link

Where is the name “bw-rule-2” defined?

| username: Ming | Original post link

That means a data source cannot have multiple rules because it doesn’t make much sense, so they didn’t implement it that way. At the time, I saw on the official website that they directly wrote a bw-rule-2, but didn’t specify where it was defined, and they didn’t write the name in the second data source either. I thought it was possible for one source to correspond to multiple rules, but I didn’t consider the significance of writing two.

| username: dba-kit | Original post link

This should actually be used when merging sharded databases and tables. Multiple upstream MySQL sources can use different block-allow-lists (but each source can only choose one of them).

| username: Ming | Original post link

Okay, thank you, got it.

| username: 舞动梦灵 | Original post link

The two whitelist rules are basically set for different databases of the same upstream data source. You can directly copy the original rule, change the name, and then add different rules to the corresponding database, or write multiple definition rules under the same rule to specify different databases or tables.

| username: Ming | Original post link

If I want to perform a filtering operation on one schema’s table and a specific operation on another schema’s table in the upstream MySQL database, do I need to set up two data sources?

| username: Ming | Original post link

I want to operate like this, but how can I define two rule names for the same data source? This was my initial question.

| username: 舞动梦灵 | Original post link

Under the same data source, you can write an additional rule. Writing multiple rules is definitely for different databases or tables, just write it like this. As you mentioned, one is for filtering, and the other is for specifying operations. You can try this, and modify the rule content according to your needs.

mysql-instances:
    - source-id: "test1"
      block-allow-list:  "bw-rule-1","bw-rule-2"
      filter-rules: ["filter1","filter2"]
      meta:
        binlog-name: "mysql-db40-bin.000080"
        binlog-pos: 423320383
block-allow-list:
  bw-rule-1:
    do-dbs: ["db_name1"]
    do-tables:
    - db-name: "*"
      tbl-name: "*"
  bw-rule-2:
    do-dbs: ["db_name2"]
    do-tables:
    - db-name: "*"
      tbl-name: "*"
filters:
  filter1:
     schema-pattern: "db_name1"
     events: ["drop database","truncate table", "drop table","delete","create index","drop index"]
     sql-pattern: ["^CREATE\\s+DEFINER","^DROP\\s+EVENT"]
     action: Ignore
  filter2:
     schema-pattern: "db_name2"
     events: ["drop database","truncate table", "drop table","delete","create index","drop index"]
     action: Ignore
| username: 洪七表哥 | Original post link

Learned.

| username: TiDBer_QYr0vohO | Original post link

Multiple rules can be defined within one rule name.

| username: Ming | Original post link

May I ask if block-allow-list: “bw-rule-1”, “bw-rule-2” has been tested and is feasible?

| username: 舞动梦灵 | Original post link

I haven’t divided it so finely before; I usually use a single rule for all databases. If this doesn’t work, then you can write two source-ids for the same data source:

mysql-instances:
  - source-id: "test1" 
    block-allow-list: "bw-rule-1" 
    filter-rules: ["filter1"]
  - source-id: "test1"  
    block-allow-list: "bw-rule-2" 
    filter-rules: ["filter2"]

Or write two data sources for the same IP address:

mysql-instances:
  - source-id: "test1" 
    block-allow-list: "bw-rule-1" 
    filter-rules: ["filter1"]
  - source-id: "test2"  
    block-allow-list: "bw-rule-2" 
    filter-rules: ["filter2"]

You can try both. I haven’t tested multiple whitelists for a single data source, but the latter two methods will definitely work. Use whichever is more convenient for you.

| username: 逍遥_猫 | Original post link

You can try adding square brackets.

| username: Ming | Original post link

Square brackets don’t work.

| username: Ming | Original post link

Hmm, I’ll try to see if two source-ids can work with the same data source.