Unable to Connect to Server Using Flink CDC Despite Network Connectivity

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用Flink CDC 无法连接服务器,但是网络畅通

| username: 天下第一帅

To improve efficiency, please provide the following information. A clear problem description can help solve the issue faster:

[Overview] Scenario + Problem Overview

[Application Framework and Development Adaptation Business Logic]
Dependencies:

Flink version 1.14.4

<dependency>
    <groupId>com.ververica</groupId>
    <artifactId>flink-connector-tidb-cdc</artifactId>
    <!-- The dependency is available only for stable releases, SNAPSHOT dependency need build by yourself. -->
    <version>2.2.0</version>
</dependency>

[Background] Actions taken

[Phenomenon] Business and database phenomena

Hint
Using Flink CDC’s TiDB, the following error was encountered when connecting:
10:56:25,283 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,283 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,284 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,284 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,323 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,324 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,325 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,326 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,328 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,328 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,328 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable
10:56:25,329 WARN org.tikv.common.region.StoreHealthyChecker - store [tidb-default-tikv-0.tidb-default-tikv-peer.component.svc:20160] is not reachable

org.tikv.shade.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 4.999980801s. [remote_addr=tidb-default-pd-0.tidb-default-pd-peer.component.svc/10.224.2.64:2379]
However, telnet can connect, and there are no network issues.

org.tikv.shade.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 4.999962518s. [remote_addr=tidb-default-pd-0.tidb-default-pd-peer.component.svc/10.224.2.64:2379]
    at org.tikv.shade.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:287) ~[tikv-client-java-3.2.0.jar:?]
    at org.tikv.shade.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:268) ~[tikv-client-java-3.2.0.jar:?]
    at org.tikv.shade.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:175) ~[tikv-client-java-3.2.0.jar:?]
    at org.tikv.kvproto.PDGrpc$PDBlockingStub.getMembers(PDGrpc.java:1868) ~[tikv-client-java-3.2.0.jar:?]
    at org.tikv.common.PDClient.getMembers(PDClient.java:443) ~[tikv-client-java-3.2.0.jar:?]
    at org.tikv.common.PDClient.tryUpdateLeader(PDClient.java:565) ~[tikv-client-java-3.2.0.jar:?]
    at org.tikv.common.PDClient.lambda$initCluster$15(PDClient.java:730) ~[tikv-client-java-3.2.0.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_291]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_291]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_291]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_291]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]

Please advise. Does TiDB require any additional operations or services to be installed?

[Problem] Current issue encountered

[Business Impact]

[TiDB Version]

[Attachments] Relevant logs and monitoring

| username: Billmay表妹 | Original post link

[Best Practices] Flink + TiDB/TiCDC

Flink + TiDB/TiCDC - Column Articles

Contributor: @懂的都懂

Flink Best Practices: Streaming TiDB Data into Flink via TiCDC

Flink + TiDB, Experience the Beauty of Real-time Data Warehouse

Flink Best Practices: Using Canal to Sync MySQL Data to TiDB

Contributor: @majiajue

TiDB + FLINK Real-time Computing

Contributor: @luzizhou

End-to-End Real-time Computing: TiDB + Flink Best Practices

TiFlink: Implementing Strongly Consistent Materialized Views with TiKV and Flink

Contributor: @Billmay

TiDB and Flink Jointly Release Best Practices White Paper for Real-time Data Warehouse

Contributor: @fancycrabtree

Flink on TiDB - Convenient and Reliable Real-time Data Business Support

NetEase Games Flink on TiDB Real-time Data Business Practices

Flink + TiDB/TiCDC - Technical Q&A

How to Improve Flink’s Write Speed to TiDB

Flink Best Practices: MySQL, TiDB Data Sync Test Plan

Garbage Characters Issue When Flink CDC Reads TiDB Database Data

[FAQ] Flink Cluster Deployment Worker Startup Failure

When Combining TiDB with Flink, Does Flink Still Run Based on HDFS?

Error When Flink SQL Reads Existing Table

Flink Best Practices: Deploying High Availability Flink Standalone Cluster from Scratch

Can TiKV Serve as Flink’s State Backend?

Abnormal Characters When Syncing to Kafka via TiCDC

How to Configure Multiple TiDB-server Addresses for Flink Writing to TiDB

Parsing Data Sent to Kafka by TiCDC with Flink

Efficiently Writing Real-time Data to TiDB with Flink

Data Loss When Flink SQL Reads TiDB Data

Are There Any Documentation Resources for Building Real-time Data Warehouses Using TiDB with Flink?

Error When Flink Uses JDBC to Continuously Write to TiDB in Batch Mode

How to Maintain Long Connections to TiDB When Flink Real-time Data Ingestion via JDBC

You can take a look at these.

| username: 天下第一帅 | Original post link

Hello, do I need to enable the CDC service for TiDB?

| username: Min_Chen | Original post link

Hello,

Could you provide your configuration details? What structure and configuration items are you using? Also, the table creation statement for TiDB CDC table.

It seems like you are experiencing connection timeouts, which likely indicates network issues. If the TiDB cluster is deployed in K8S, the application should also be running in K8S.