In TiDB 4.0.x version, unable to read data when pulling full and incremental data through tikv-client-java

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 在tidb 4.0.x版本,通过tikv-client-java拉取全量以及增量数据时无法读取数据

| username: TiDBer_mEYNZXIU

[TiDB Usage Environment] Production Environment
[TiDB Version] 4.0.6–4.0.10
[tikv-client-java Version] 3.2.0
[Reproduction Path] Using tidb-cdc-connector to pull full data
[Encountered Problem: Phenomenon and Impact] Unable to read data

Background:
I want to import data from TiDB to other media through flink-cdc. During the reading process, Meta information can be accessed, but the actual kv cannot be obtained from the store. Specifically, it can be seen in the sdk: tikv-client-java’s RegionStoreClient’s scan method.


The request shows the information of accessing the specific store, but the final response has no return value.

There are no errors, guessing whether it is related to timeout when fetching data from the store, or the link being half-closed; or it is a version issue (currently the company has no intention to upgrade the version).

| username: xfworld | Original post link

  1. The Flink CDC TiDB connector is not produced by the TiDB community.
  2. I suggest you first organize the relevant versions and determine if the issue is caused by version mismatches.
  3. TiDB 4.x is no longer a supported version; it is recommended to upgrade to 6.X.
| username: Fly-bird | Original post link

Use TiCDC to synchronize to Kafka and then consume with Flink.

| username: TiDBer_mEYNZXIU | Original post link

However, during the troubleshooting, it was found that tikv-client-java 3.2.0 could not retrieve KV from the TiKV store when performing a scan action. The flink-connector was merely calling the API.

| username: 有猫万事足 | Original post link

Are you using this?

tidb-cdc * TiDB: 5.1.x, 5.2.x, 5.3.x, 5.4.x, 6.0.0 JDBC Driver: 8.0.27

If so, the homepage doesn’t mention support for TiDB 4.0, right? :sweat_smile:

| username: TiDBer_mEYNZXIU | Original post link

However, I found that version 4.0.10 can be used normally during my testing.

| username: 有猫万事足 | Original post link

So, has this problem been resolved?
It suddenly started working again.

| username: xfworld | Original post link

Ignore version requirements :yum: :yum: :yum: :yum:

| username: Billmay表妹 | Original post link

Try upgrading the version to solve your problem.

| username: TiDBer_mEYNZXIU | Original post link

The main reason is that the company has a large number of tables in versions 4.0.6 and 4.0.8, and the upgrade cost is something the DBA is unwilling to accept, so they are trying to continue finding solutions for these versions.

| username: Billmay表妹 | Original post link

This issue has dependencies:

The Flink CDC TiDB connector is not produced by the TiDB community. If you are not planning to upgrade TiDB to solve the problem, you may need to seek a solution from the other community.

| username: 有猫万事足 | Original post link

The DBA’s perspective is understandable.

Regarding the flink-cdc-connectors tool, which hasn’t promised support, it’s best if it works. If it doesn’t, you can either raise a feature request on the corresponding GitHub project and wait for community support, or see if you can understand the code and contribute to the ecosystem yourself. These are the two options.

From TiDB’s perspective, the community can provide support through TiCDC:

This can be used from version 4.0.6 onwards.

Back to your initial question:

Importing data from TiDB to other media

I don’t think it’s necessary to be too concerned about which tool to use. Would it be very difficult to use TiCDC?

| username: TiDBer_mEYNZXIU | Original post link

It’s possible, I mean it’s possible… that no team is willing to take on the additional operational work of a new component :joy:

| username: 有猫万事足 | Original post link

Agreed.

The flink-cdc-connectors team might be thinking the same way. That’s why they don’t support TiDB 4.0. You can try urging them.

| username: xfworld | Original post link

Starting from version 4.X, it is recommended to use the binlog component instead of ticdc :yum:

The downstream can be any file system, which can parse the binlog through scheduling methods to obtain changes in real-time.
It can also be connected to Kafka, which is more complex.

| username: TiDBer_mEYNZXIU | Original post link

Which specific binlog component is this?

| username: xfworld | Original post link

If there are a large number of DDLs, using binlog might have some issues, so be aware of this…

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.