Issues with Kafka Synchronization Authentication in TiCDC

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC同步认证的kafka问题

| username: seiang

[TiDB Usage Environment] Production Environment / Test / Poc
[TiDB Version] v6.5.1
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Problem Phenomenon and Impact]
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot / Logs / Monitoring]

Using TiCDC to synchronize authenticated Kafka, encountering Kafka authentication failure issue,

tiup ctl:v6.5.1 cdc changefeed create --pd=http://xxxxx:2379 --sink-uri=“kafka://xxxxx:9092/topic_name?kafka-version=2.7.1&sasl-user=tidb_cdc&sasl-password=xxxxxx&sasl-mechanism=PLAIN&max-message-bytes=1048576” --changefeed-id=“kafka-task” --sort-engine=“unified” --config=./changefeed.toml

Special Notes:

  1. Kafka password does not contain any special characters, only a combination of numbers and letters.
  2. Kafka authentication method is SASL/PLAIN.

Error: new sarama producer: Cluster authorization failed.

Logs are as follows:
[2024/01/04 12:21:44.572 +08:00] [INFO] [base_client.go:378] [“[pd] switch leader”] [new-leader=http://XXXX:2379] [old-leader=]
[2024/01/04 12:21:44.572 +08:00] [INFO] [base_client.go:105] [“[pd] init cluster id”] [cluster-id=7073366411188089559]
[2024/01/04 12:21:44.572 +08:00] [INFO] [client.go:702] [“[pd] tso dispatcher created”] [dc-location=global]
[2024/01/04 12:21:44.585 +08:00] [INFO] [sink.go:313] [“succeed to parse parameter from sink uri”] [protocol=default] [txnAtomicity=]
[2024/01/04 12:21:44.903 +08:00] [INFO] [sink.go:313] [“succeed to parse parameter from sink uri”] [protocol=default] [txnAtomicity=]
[2024/01/04 12:21:44.927 +08:00] [INFO] [client.go:783] [“[pd] stop fetching the pending tso requests due to context canceled”] [dc-location=global]
[2024/01/04 12:21:44.927 +08:00] [INFO] [client.go:720] [“[pd] exit tso dispatcher”] [dc-location=global]
[2024/01/04 12:21:44.927 +08:00] [INFO] [middleware.go:48] [/api/v2/changefeeds] [status=500] [method=POST] [path=/api/v2/changefeeds] [query=] [ip=xxxxx] [user-agent=Go-http-client/1.1] [client-version=v6.5.1] [error=“[CDC:ErrKafkaNewSaramaProducer]new sarama producer: Cluster authorization failed.”] [errorVerbose=“[CDC:ErrKafkaNewSaramaProducer]new sarama producer: Cluster authorization failed.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq.NewKafkaDMLSink\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq/kafka_dml_sink.go:72\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/factory.New\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/factory/factory.go:68\ngithub.com/pingcap/tiflow/cdc/sink.Validate\n\tgithub.com/pingcap/tiflow/cdc/sink/validator.go:63\ngithub.com/pingcap/tiflow/cdc/api/v2.APIV2HelpersImpl.verifyCreateChangefeedConfig\n\tgithub.com/pingcap/tiflow/cdc/api/v2/api_helpers.go:240\ngithub.com/pingcap/tiflow/cdc/api/v2.(*OpenAPIV2).createChangefeed\n\tgithub.com/pingcap/tiflow/cdc/api/v2/changefeed.go:81\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.ForwardToOwnerMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:95\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.ErrorHandleMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:64\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.LogMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:38\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.CheckServerReadyMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:103\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991\nruntime.goexit\n\truntime/asm_amd64.s:1594”] [duration=371.528001ms]

2
| username: wangccsy | Original post link

What is Kafka?

3
| username: 像风一样的男子 | Original post link

Is the Kafka version correct? Can you confirm that it is 2.7.1?

4
| username: Daniel-W | Original post link

Is the topic name written correctly?

5
| username: seiang | Original post link

Make sure the topic name is correct.

6
| username: seiang | Original post link

Kafka version confirmed to be 2.7.1

7
| username: jiaxin | Original post link

We use Kafka version 2.3 for CDC synchronization, without authentication, and haven’t encountered this issue :rofl:

8
| username: dba远航 | Original post link

Check whether the password is correct, whether authorization is granted, and whether the firewall is disabled, etc.

9
| username: 小龙虾爱大龙虾 | Original post link

Try using lowercase “plain”? (Just a guess)

10
| username: FutureDB | Original post link

Check if it is a Kafka user permission issue. TiCDC has certain permission requirements when connecting to Kafka through ACL:

The minimum set of permissions required for TiCDC to work properly is as follows:

  1. Create and Write permissions for the Topic resource type.
  2. DescribeConfigs permission for the Cluster resource type.

Reference:
TiCDC Authentication and Authorization with Kafka

You can check: TiCDC 无法接入kafka - TiDB 的问答社区.

11
| username: seiang | Original post link

It’s not because of that reason. Our Kafka authentication method is SASL/PLAIN, and it’s not a permissions issue either. I wrote the code myself, using the same topic and the same username and password, and there were no problems with authentication and data writing.

12
| username: 烂番薯0 | Original post link

Advanced…

13
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.