TiKV Error cdc initialize fail: Other error [components/cdc/src/initializer.rs:124]: scan canceled

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiKV 报错 cdc initialize fail: Other error [components/cdc/src/initializer.rs:124]: scan canceled

| username: mxd-321

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.0
[Encountered Problem: Phenomenon and Impact] The source table has more than 7000 regions, and the downstream is obtaining CDC events through tikv-client-java, which is reporting errors like crazy: cdc initialize fail: Other error [components/cdc/src/initializer.rs:124]: scan canceled. Can someone help take a look?
[Attachment: Screenshot/Log/Monitoring]

image
image1380×864 581 KB

2
| username: mxd-321 | Original post link

I understand that reporting not_leader should be normal, but are the other errors abnormal? Can any experts take a look?

3
| username: songxuecheng | Original post link

Check the logs of the PD leader.

4
| username: mxd-321 | Original post link

I checked, and there are no errors in the PD logs, only some warnings.

5
| username: songxuecheng | Original post link

The image you provided is not accessible. Please provide the text you need translated.

6
| username: mxd-321 | Original post link

Master, what does this mean here? How can I fix it?

7
| username: liuis | Original post link

Is there a problem with the network?

8
| username: songxuecheng | Original post link

What components are associated with the two IPs above? Please provide the display.

9
| username: mxd-321 | Original post link

Boss, this is display.

10
| username: songxuecheng | Original post link

Check if the network for ports 188 to 190 is normal, as it is a mixed deployment. Also, check the disk space, I/O, and memory for any abnormalities.

11
| username: mxd-321 | Original post link

Boss, what you mentioned is all fine. These two machines are both deployed in the same local area network.

12
| username: songxuecheng | Original post link

The error in the logs indicates a network issue or firewall, with the port being closed.

13
| username: mxd-321 | Original post link

Alright, the network should be fine, and our machine hasn’t enabled the firewall either.

image
image926×310 6.7 KB

14
| username: Lucien-卢西恩 | Original post link

@mxd-321 Check when the earliest occurrence of “cdc initialize fail: Other error [components/cdc/src/initializer.rs:124] scan canceled” log is, and provide the 100 lines of logs before and after it.

15
| username: mxd-321 | Original post link

This is a log from a TiKV node. This error has been occurring for a long time, and the downstream tikv-client-java keeps stopping and starting. My initial judgment is that the TiDB version is relatively high and is incompatible with tikv-client-java. The previously tested version 5.1.4 had no issues.

16
| username: Lucien-卢西恩 | Original post link

Are you currently using TiCDC with your TiKV cluster? You didn’t have this issue before, right? You might want to try the latest version of tikv-client-java.

17
| username: mxd-321 | Original post link

We didn’t use TiCDC; we wrote our own component, which includes tikv-client-java. The latest version of tikv-client-java no longer has the cdcClient component.
May I ask if there are any other ways to use Java to get incremental data from TiKV?

18
| username: Lucien-卢西恩 | Original post link

Currently, there is no solution for this in the product. You can discuss this topic in the issues section on Github: GitHub - pingcap/tiflow: This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)