TiDB Fails to Write Data to Kafka via Drainer

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb通过 drainer写数据到kafka不成功

| username: kuweilong666

[TiDB Usage Environment] Production Environment
[TiDB Version] 4.0.0
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

drainer configuration
image
After TiDB data changes, the data is not written to Kafka
Client background metadata update: kafka: no specific topics to update metadata

| username: Billmay表妹 | Original post link

Based on the provided information, the following reasons might be causing the data not to be written to Kafka:

  1. The Kafka topic is not correctly configured or not created. Please ensure that the Kafka topic has been correctly created and that the topic name is correctly configured in the Drainer configuration file.

  2. The Kafka address in the Drainer configuration file is incorrect. Please ensure that the Kafka address is correctly configured in the Drainer configuration file.

  3. Kafka version incompatibility. Please ensure that the Kafka version used by Drainer is compatible with the Kafka cluster version.

  4. Some nodes in the Kafka cluster are unavailable. Please check if all nodes in the Kafka cluster are running normally.

  5. The Drainer logs might provide more information about the issue. Please check the Drainer logs for more details about the problem.

Additionally, based on the provided information, the “Client background metadata update: kafka: no specific topics to update metadata” error might be due to not specifying the specific topic to update. Please ensure that the topic name to be updated is correctly specified in the Drainer configuration file.

If the above methods do not resolve the issue, please provide more detailed information to better assist in solving the problem.

| username: kuweilong666 | Original post link

[2023/05/31 16:56:17.980 +08:00] [INFO] [broker.go:168] [“[sarama] Failed to connect to broker localhost:9092: dial tcp [::1]:9092: connect: connection refused\n”]
[2023/05/31 16:56:17.980 +08:00] [INFO] [async_producer.go:971] [“[sarama] producer/broker/0 state change to [closing] because dial tcp [::1]:9092: connect: connection refused\n”]
[2023/05/31 16:56:17.980 +08:00] [INFO] [async_producer.go:578] [“[sarama] producer/leader/test/0 state change to [retrying-75]\n”]
[2023/05/31 16:56:17.980 +08:00] [INFO] [async_producer.go:588] [“[sarama] producer/leader/test/0 abandoning broker 0\n”]
[2023/05/31 16:56:17.980 +08:00] [INFO] [async_producer.go:717] [“[sarama] producer/broker/0 input chan closed\n”]
[2023/05/31 16:56:17.980 +08:00] [INFO] [async_producer.go:801] [“[sarama] producer/broker/0 shut down\n”]
[2023/05/31 16:56:18.480 +08:00] [INFO] [client.go:772] [“[sarama] client/metadata fetching metadata for [test] from broker 192.168.10.44:9092\n”]
[2023/05/31 16:56:18.482 +08:00] [INFO] [async_producer.go:711] [“[sarama] producer/broker/0 starting up\n”]
[2023/05/31 16:56:18.483 +08:00] [INFO] [async_producer.go:727] [“[sarama] producer/broker/0 state change to [open] on test/0\n”]
[2023/05/31 16:56:18.483 +08:00] [INFO] [async_producer.go:570] [“[sarama] producer/leader/test/0 selected broker 0\n”]
[2023/05/31 16:56:18.483 +08:00] [INFO] [async_producer.go:594] [“[sarama] producer/leader/test/0 state change to [flushing-75]\n”]
[2023/05/31 16:56:18.483 +08:00] [INFO] [async_producer.go:616] [“[sarama] producer/leader/test/0 state change to [normal]\n”]
[2023/05/31 16:56:18.483 +08:00] [INFO] [broker.go:168] [“[sarama] Failed to connect to broker localhost:9092: dial tcp [::1]:9092: connect: connection refused\n”]
[2023/05/31 16:56:18.484 +08:00] [INFO] [async_producer.go:971] [“[sarama] producer/broker/0 state change to [closing] because dial tcp [::1]:9092: connect: connection refused\n”]
[2023/05/31 16:56:18.484 +08:00] [INFO] [async_producer.go:578] [“[sarama] producer/leader/test/0 state change to [retrying-76]\n”]
[2023/05/31 16:56:18.484 +08:00] [INFO] [async_producer.go:588] [“[sarama] producer/leader/test/0 abandoning broker 0\n”]
[2023/05/31 16:56:18.484 +08:00] [INFO] [async_producer.go:717] [“[sarama] producer/broker/0 input chan closed\n”]
[2023/05/31 16:56:18.484 +08:00] [INFO] [async_producer.go:801] [“[sarama] producer/broker/0 shut down\n”]

| username: kuweilong666 | Original post link

“TiDB 5.0 version has already GA, it is recommended to upgrade to the latest version for more features and performance improvements.”

| username: kuweilong666 | Original post link

The binlog is not enabled.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.