How to Check Incomplete Data and Latency When Syncing TiDB with MySQL

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb同步mysql数据，数据不全，延迟如何查看

| username: 初来乍到

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the problem occurred
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]
I am a newbie, first time encountering TiDB. The previous person in charge just left and didn’t leave any TiDB documentation. No one knows how to handle TiDB. Today, business queries reported that the data in the TiDB tables is incomplete. The data is synchronized from MySQL. Could any experts advise on how to analyze this issue? I am a beginner with TiDB and just started learning. Apologies for the basic question.

2
| username: zhanggame1 | Original post link

First, check how it was synchronized. The TiDB ecosystem tool is DM, but you can also use other software.

3
| username: 像风一样的男子 | Original post link

Oh, it looks like a big pit.

4
| username: 气死人的萌新 | Original post link

The synchronized data, can we take a look at the DM configuration file?

5
| username: 像风一样的男子 | Original post link

First, check out the common issues with DM. Troubleshoot them one by one.

6
| username: tony5413 | Original post link

Specific problems require specific analysis. Identify which tables are incomplete and examine the extent of the incompleteness. Then check if there are any migration logs left.

7
| username: Miracle | Original post link

This is a situation where risks and opportunities coexist.

8
| username: tidb菜鸟一只 | Original post link

If it is synchronized through DM, first check the status of the DM task.

9
| username: 大飞哥online | Original post link

You can check the status of DM using the command tiup dmctl --master-addr <master-address> query-status.

10
| username: 大飞哥online | Original post link

The DM task “sbtest” is running, and its data source is “mysql-replica-01”.

11
| username: Fly-bird | Original post link

First, determine whether it is real-time synchronization or one-time synchronization. If the data is indeed incomplete, it is recommended to re-synchronize the full data and then perform real-time synchronization.

12
| username: Kongdom | Original post link

If there are no handover documents, it can indeed be problematic.

  1. First, confirm the synchronization method.
  2. Compare the full data by time point to eliminate potential risks.
  3. Check the TiDB version, running status, cluster topology, and other information.
13
| username: andone | Original post link

What state does DM run in?

14
| username: Jellybean | Original post link

This statement suggests that there is an inconsistency in the existing baseline data. You can consider using the official sync_diff_inspector tool to verify and compare the baseline data. Its use case is to check the consistency of two sets of data in MySQL/TiDB, and it can also output repair SQL to fill in the missing data where there are differences.

For detailed operations, you can refer to the official documentation:

15
| username: Soysauce520 | Original post link

Are you using the DM tool?

16
| username: 初来乍到 | Original post link

Problem solved, thank you.

17
| username: 初来乍到 | Original post link

Thank you, much appreciated.

18
| username: 初来乍到 | Original post link

Don’t mention it. Consulting with senior employees requires a fee.

19
| username: Kongdom | Original post link

:thinking: How did you solve it? Can you share the solution?

20
| username: shuyu_zhihui | Original post link

Doesn’t this tool generate multiple SQL files? How do you execute so many SQL files in bulk?