Timing for Initiating DM Incremental Data Verification

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: DM增量数据校验的启动时机

| username: TiDBer_lpPVewBC

[Test Environment for TiDB] Testing
[TiDB Version] 6.5.0
[Question] Our business development has a real-time verification requirement during data migration. We found that the DM tool in the TiDB documentation has a real-time verification function for incremental data https://docs.pingcap.com/zh/tidb/stable/dm-continuous-data-validation. In the process description of the validation worker, it is mentioned: When the validation worker accumulates a certain number of rows or after a certain time interval, the validation worker queries the current data in the downstream database based on the primary key information of these rows and compares it with the expected data of the changed rows. The comparison trigger conditions include the number of rows and the time interval. Can these conditions be customized through the configuration file, or are they already configured in the source code and cannot be modified?

2
| username: Lucien-卢西恩 | Original post link

This check can only modify the number of concurrent workers and the check mode. If you need to set specific conditions for the check, you can use sync-diff-inspector 用户文档 | PingCAP 文档中心

3
| username: TiDBer_lpPVewBC | Original post link

Okay, I have also used sync-diff-inspector, but it seems that online verification cannot be achieved.

4
| username: TiDBer_lpPVewBC | Original post link

May I ask if the trigger condition for the validation worker’s data validation is a fixed value, that is, when the number of changed data rows in the upstream database reaches a specified value or the changed data rows have not been validated within a specific time? Or is it dynamically adjusted according to specific situations? In the simulated environment test, it seems to trigger validation activities based on fixed values.

5
| username: D3Hunter-pingcap | Original post link

Yes, currently these two values are 100,000 rows or 10 seconds, and configuration is not open.

6
| username: TiDBer_lpPVewBC | Original post link

Okay :grin:

7
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.