Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: ticdc手工删除信息不同步到下游,业务删除信息同步到下游,如何做到
【TiDB Usage Environment】Production Environment / Testing / PoC
【TiDB Version】tidb 6.5
【Reproduction Path】ticdc condition filtering
ignore-delete-value-expr = “name = ‘john’” # Filter out delete DMLs that contain the condition name = ‘john’
I set ignore-delete-value-expr = “1 = 1’” #
In my cdc configuration file, I configured the above filtering condition to distinguish between manual deletions that are not synchronized downstream and actual business deletions that are synchronized downstream.
delete from t_table where no=‘92171802708161882’ and 1 = 1
The above manual deletion statement was synchronized to Kafka.
【Encountered Problem: Problem Phenomenon and Impact】
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】
You have an extra quotation mark here?
It’s useless. CDC captures the change logs of TiKV. CDC cannot see the original SQL you executed, and its filtering is based on the value of the data.
This 1=1 actually has no effect and will be optimized out during the logical optimization phase. The ignore-delete-value-expr must be within the range of the column that is actually to be deleted in order to take effect.
I also think that “where 1=1” is useless, it’s just to mark the statement, the specific value is useful. Moreover, after using the expression to filter, batch deletion of unsynchronized data will cause CDC delay.
I am seriously testing the functionality I need to implement.
CDC captures the change data from TiKV and only recognizes row data changes. It cannot distinguish whether the change was made manually or by business operations.
Therefore, the conclusion is that regardless of the source, as long as a row of data is changed, TiCDC will synchronize these changes to the downstream.
The underlying mechanism for CDC to obtain changes is by retrieving key-value pairs from TiKV, not through the binlog of the TiDB server. Therefore, it cannot distinguish differences in your SQL.