The ignore-txn-start-ts parameter in the configuration file does not take effect

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 配置文件中配置ignore-txn-start-ts参数不生效

| username: HACK

【TiDB Usage Environment】Production, Testing, Research
【TiDB Version】
【Encountered Problem】
Using the ignore-txn-start-ts parameter in the configuration file to skip specified operations did not take effect.

  1. Stop CDC replication

  2. Insert a record with ID=333 in the upstream (at this point, the downstream table does not have a record with ID=333)
    insert into cdcdb.test5 values(333)

  3. Check the current database’s TSO
    cdc cli --pd=http://xxx.xxx.xxx.xxx tso query
    434708167235993601

  4. Edit the synchronization configuration file to add ignore-txn-start-ts
    vi cdc.toml
    ignore-txn-start-ts = [434708167235993601]

  5. Start CDC replication
    $ tiup ctl:v4.0.16 cdc changefeed update --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=“cdc2” --config cdc.toml
    Starting component ctl: /home/tidb/.tiup/components/ctl/v4.0.16/ctl cdc changefeed update --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=cdc2 --config cdc.toml
    Diff of changefeed config:
    {Type:create Path:[Config Filter IgnoreTxnStartTs 0] From: To:434708167235993601}
    Could you agree to apply changes above to changefeed [Y/N]
    y
    Update changefeed config successfully! Will take effect only if the changefeed has been paused before this command
    ID: cdc2
    Info: {“sink-uri”:“tidb://root:admin@10.19.248.128:4000/”,“opts”:{“_changefeed_id”:“cli-verify”},“create-time”:“2022-07-19T11:00:21.200455131+08:00”,“start-ts”:434686528597262343,“target-ts”:0,“admin-job-type”:1,“sort-engine”:“unified”,“sort-dir”:“”,“config”:{“case-sensitive”:true,“enable-old-value”:true,“force-replicate”:true,“check-gc-safe-point”:true,“filter”:{“rules”:[“.”],“ignore-txn-start-ts”:[434708167235993601]},“mounter”:{“worker-num”:16},“sink”:{“dispatchers”:null,“protocol”:“default”},“cyclic-replication”:{“enable”:false,“replica-id”:0,“filter-replica-ids”:null,“id-buckets”:0,“sync-ddl”:false},“scheduler”:{“type”:“table-number”,“polling-time”:-1}},“state”:“stopped”,“history”:null,“error”:null,“sync-point-enabled”:false,“sync-point-interval”:600000000000,“creator-version”:“v4.0.16”}

  6. Check the changefeed information, and there is ignore-txn-start-ts related information
    tiup ctl:v4.0.16 cdc changefeed query --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=cdc2
    Starting component ctl: /home/tidb/.tiup/components/ctl/v4.0.16/ctl cdc changefeed query --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=cdc2
    {
    “info”: {
    “sink-uri”: “tidb://root:admin@10.19.248.128:4000/”,
    “opts”: {
    “_changefeed_id”: “cli-verify”
    },
    “create-time”: “2022-07-19T11:00:21.200455131+08:00”,
    “start-ts”: 434686528597262343,
    “target-ts”: 0,
    “admin-job-type”: 0,
    “sort-engine”: “unified”,
    “sort-dir”: “”,
    “config”: {
    “case-sensitive”: true,
    “enable-old-value”: true,
    “force-replicate”: true,
    “check-gc-safe-point”: true,
    “filter”: {
    “rules”: [
    “.”
    ],
    “ignore-txn-start-ts”: [
    434708167235993601
    ]
    },

  7. Check the downstream for ID=333 and find that the record has been synchronized.

Question: Why is this happening? It seems that ignore-txn-start-ts did not take effect.

【Reproduction Path】What operations were performed to encounter the problem
【Problem Phenomenon and Impact】

【Attachments】

Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.

2
| username: songxuecheng | Original post link

According to this operation, stop the old one and start the new one.

3
| username: HACK | Original post link

Is it certain that a new changefeed must be created? I modified the original changefeed.

4
| username: songxuecheng | Original post link

Need to rebuild one.

5
| username: HACK | Original post link

Create a new changefeed, then synchronize the configuration file, and add ignore-txn-start-ts vi cdc.toml ignore-txn-start-ts = [434708167235993601]
Is that correct?

6
| username: HACK | Original post link

Why do I feel like this is a bit repetitive?

7
| username: songxuecheng | Original post link

In the new task configuration file, skip the specified transaction, and set start-ts to the checkpoint-ts of the old task to ensure no data loss.

8
| username: HACK | Original post link

Got it, give it a try.

9
| username: HACK | Original post link

I am now conducting another test. An upstream drop table operation is executed, but this table does not exist downstream, causing the replication to fail. After checking the current checkpoint, I increment the start-ts checkpoint by 1 and update the changefeed. However, it prompts that this start-ts is ignored. What is the reason for this?

$ cdc cli changefeed query  --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=testcdc | grep "checkpoint-ts"
    "checkpoint-ts": 434709261042057217,
    
$ tiup ctl:v4.0.16 cdc changefeed update --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id="testcdc" --start-ts 434709703161020418 --config cdc.toml
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.16/ctl cdc changefeed update --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=testcdc --start-ts 434709703161020418 --config cdc.toml
[2022/07/20 11:21:08.637 +08:00] **[WARN] [client_changefeed.go:552] ["this flag cannot be updated and will be ignored"] [flagName=start-ts]**
changefeed config is the same with the old one, do nothing
10
| username: songxuecheng | Original post link

Skip the erroneous steps as before, do not modify directly.

11
| username: HACK | Original post link

To create a new synchronization task, how should I configure the ignore-txn-start-ts in the configuration file and the start-ts specified when creating the changefeed?

Should it be set like this:

ignore-txn-start-ts: checkpoint of the changefeed

start-ts: checkpoint-ts of the changefeed + 1

12
| username: songxuecheng | Original post link

The error log contains txn ts.

13
| username: HACK | Original post link

After checking the cdc.log log, I didn’t find any relevant ts prompt content when there was an error.

14
| username: neilshen | Original post link

You can search for Row changed event ignored in the TiCDC logs. If it exists, it indicates that ignore-txn-start-ts has been correctly configured and successfully filtered out the related event.

15
| username: HACK | Original post link

Here is my testing process:

  1. Create a table TEST upstream and insert data with id=1.

  2. Manually delete the automatically synchronized TEST table downstream.

  3. Check the CDC replication task failure, indicating that TEST does not exist.

In this scenario, how can I skip this error and let the replication continue?

I did not find relevant ts information in cdc.log. If I could see the ts information at this time, I could skip it.

Thank you.

16
| username: neilshen | Original post link

This is a very good suggestion. Currently, there is no similar log. An issue has been created Log start-ts in log when TiCDC MySQL sink fails to execute DML/DDL · Issue #6460 · pingcap/tiflow (github.com)

17
| username: HACK | Original post link

The questions I have now are:

  1. If I encounter a DDL execution error, I can handle it as suggested on the official website: find the current checkpoint-ts value of the synchronization task, add 1 to it, and then create a new synchronization task.

    DDL
    DDL805×461 24.8 KB

  2. If I encounter other issues, such as the scenarios mentioned by the official website

    DML
    DML811×700 36.4 KB

I can find the checkpoint-ts value from the synchronization task, but how do I know the ts value at the point where the error occurred (I don’t know how to configure the value for ignore-txn-start-ts), that is, I don’t know the ts value to skip, how to find it.

18
| username: neilshen | Original post link

The start-ts has been added in this PR sink(ticdc): add start ts to mysql sink and refine get tso by overvenus · Pull Request #6462 · pingcap/tiflow (github.com)

19
| username: HACK | Original post link

:+1:

20
| username: system | Original post link

This topic will be automatically closed 60 days after the last reply. No new replies are allowed.