Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: cdc报错：[CDC:ErrKafkaNewSaramaProducer]kafka:
|
username: db_user
Version: 4.0.13
Kafka version: 3.1.0
Detailed error:
[CDC:ErrKafkaNewSaramaProducer] kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
The Kafka version is specified in the sink, so it’s not the issue described in the FAQ. CDC has no logs, and I can telnet to the Kafka port.
It shouldn’t be. I tried a lower version, and it shouldn’t be an incompatibility issue. My Kafka is a single node, not sure if that has anything to do with it.
As a last resort, I switched to binlog to solve this problem, but I still don’t understand the principle. It might be that CDC doesn’t support non-cluster mode. I checked the issue and my environment, and it’s not a SASL problem.
TiCDC owner gets stuck when creating new kafka producer · Issue #3352 · pingcap/tiflow · GitHub You can check if the error is the same.
You can try upgrading to a higher version in the test environment.
I tried installing Kafka on the same machine as the local TiDB cluster, and it started normally. However, when installed elsewhere, it reported an error:
[2022-06-30 14:17:37,573] WARN [SocketServer brokerId=0] Unexpected error from /cdc_ip; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = -720899)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:103)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:447)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:397)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:913)
at kafka.network.Processor.run(SocketServer.scala:816)
at java.lang.Thread.run(Thread.java:748)
I’ll try upgrading the cluster version to see if it helps.
Buddy, I’ve tried 4.0.13, 5.0.1, 5.3.1, and 5.4.1 here, but none of them can connect. It’s very strange. Does the Kafka machine need to access a specific port of the TiDB cluster?
I understand the reason now. After I opened the 9092 port of the Kafka machine on the machine where tiup (ctl) is located, the CDC task could be created normally. Originally, I only opened the 9092 port to the CDC machine. This is actually a bit different from my understanding. I thought the ctl machine didn’t need to access Kafka’s 9092 port, only the CDC machine needed to access Kafka’s 9092 port. This is quite strange and I don’t quite understand it.
It seems that when creating a CDC task, the ctl machine needs to access Kafka’s 9092 port, and during data transmission, the CDC machine accesses Kafka’s 9092 port.
Is the issue occurring during the stage of creating a changefeed using tiup ctl? At this stage, it is tiup ctl that accesses Kafka, so it needs to have the ability to access Kafka.
Yes, but I haven’t quite filtered through the source code and haven’t seen where ctl needs to access Kafka. Could you provide the source code location, please? Additionally, I would like to ask if every ctl operation, such as adjusting configurations, starting, stopping, etc., is directly accessing Kafka through ctl, rather than through the CDC machine accessing Kafka?
The TiUP host will store the binaries of various components, and it directly calls these binaries to execute the corresponding commands. In other words, the commands are issued by the TiUP host, not by a CDC placed on some remote end.
As for accessing Kafka, this is done in tiflow/sink.go at master · pingcap/tiflow (github.com).
When executing cdc cli changefeed create --sink-uri="kafka://xxxx", the CLI will call this part of the code.
tiup ctl:vxxxx cdc is essentially executing cdc cli.
Boss, I also looked at this part when I was reading the source code before, but I didn’t see any operation connecting to Kafka. I only saw the init operation connecting to Kafka. Can you explain it in detail?
This topic was automatically closed 1 minute after the last reply. No new replies are allowed.
|
username: 
|
username: 
|
username: