TiCDC unable to connect to Kafka

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC 无法接入kafka

| username: GreenGuan

Kafka version: 2.5.1
Number of Kafka partitions: 3
TiDB version: v5.4.1

Command parameters:

tiup cdc:v5.4.1 cli changefeed create -c xxx --pd=http://xxx.xxx.xxx.xxx:xxx --sink-uri="kafka://kfkurl/topic_name?protocol=canal-json&enable-tidb-extension=true&kafka-version=2.5.1&partition-num=3&kafka-client-id=producer_name&max-message-bytes=8388608&replication-factor=3&sasl-user=xxxx&sasl-password=xxxx&sasl-mechanism=SCRAM-SHA-256" --sort-engine="unified" --config=/home/xxx/xxx/xxxx.yaml

Error:

[CDC:ErrKafkaNewSaramaProducer]new sarama producer: [CDC:ErrKafkaInvalidConfig]because TiCDC Kafka producer's `request.required.acks` defaults to -1, TiCDC cannot deliver messages when the `replication-factor` is less than `min.insync.replicas`: kafka server: The client is not authorized to send this request type.

I have checked this bug, but it was already fixed in 5.4.0:

Could you please tell me which parameter I have not set correctly?

2
| username: db_user | Original post link

I don’t think it’s the bug you mentioned. It should be this bug: Error messages in package kafka AdjustConfig() is misleading · Issue #5293 · pingcap/tiflow · GitHub. The merge time also seems to be before version 5.4.1, so it’s not clear whether it affects version 5.4.1. I think you can first create a topic in Kafka and then use CDC to synchronize, which should solve the problem.

3
| username: GreenGuan | Original post link

The bug occurred when creating a task with CDC after pre-creating a topic in Kafka. I tested it on the Kafka used for testing, and it can synchronize normally after removing the “asal”.

4
| username: db_user | Original post link

Uh, could it be that Kafka itself hasn’t enabled SASL authentication? Otherwise, it would be very strange. If authentication is enabled, how can it still synchronize normally after removing the authentication parameters?

5
| username: GreenGuan | Original post link

Oh, what I mean is
In the production environment, both Kafka and TiCDC have authentication enabled, which triggered a bug.
In the test environment, Kafka and TiCDC do not have authentication enabled, and CDC tasks can be created normally.

6
| username: Min_Chen | Original post link

Hello, what is the value of min.insync.replicas? Please confirm if sasl-user and sasl-password are correct. You can try changing sasl-mechanism=SCRAM-SHA-256 to lowercase.

7
| username: FutureDB | Original post link

I tried it, but it doesn’t work. The value after sasl-mechanism must be in uppercase.

8
| username: FutureDB | Original post link

I encountered the error “Kafka server: the client is not authorized to send this request type,” which seems similar to your issue. I spent a long time troubleshooting and eventually discovered it was a Kafka user permissions problem. TiCDC requires certain permissions to connect to Kafka via ACL:

The minimum set of permissions required for TiCDC to function properly are:
(1) Create and Write permissions for the Topic resource type.
(2) DescribeConfigs permission for the Cluster resource type.

The reason for my issue was that the Kafka user lacked Write permission for the Topic resource type and DescribeConfigs permission for the Cluster resource type.

Note: TiDB version is V5.3.2

Reference:
https://docs.pingcap.com/zh/tidb/dev/manage-ticdc

9
| username: Min_Chen | Original post link

Is the Kafka version correct?

10
| username: GreenGuan | Original post link

Correct.

11
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.