Issues with CDC using SASL to sync Kafka, causing a large number of Describe all topics requests

translator_bot · June 22, 2024, 8:16pm

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: cdc使用sasl同步kafka，出现大量Describe所有topic的问题

| username: Cloud王

[TiDB Usage Environment] Production Environment
[TiDB Version] v4.0.15, v5.4.3
[Reproduction Path]
Using SASL username and password for CDC to sync with Kafka, the creation statement is:
cdc cli changefeed create --pd=http://XXX:2379 --start-ts=XXXXX --sink-uri=“kafka://XXX:9092/topicname?partition-num=3&protocol=canal-json&replication-factor=3&sasl-user=XXX&sasl-password=XXX&sasl-mechanism=SCRAM-SHA-256” --changefeed-id=“XXX” --config=./XXX.toml

[Encountered Problem: Phenomenon and Impact]

Problem Description
After the sync task is created, Kafka logs show a large amount of the following content:
INFO Principal = User:XXX is Denied Operation = Describe from host = xxx.xxx.xxx.xxx on resource = Topic:LITERAL:XXX (kafka.authorizer.logger)

That is, the CDC node will repeatedly Describe all topics in Kafka. Currently, there are many topics in Kafka, and the large number of Describe operations is causing performance issues.
This problem occurs with sync tasks created in both v4.0.15 (production environment) and v5.4.3 (test environment).

Problem Investigation
If CDC uses Golang’s Sarama implementation and does not configure conf.Metadata.Full=false, this issue will occur:
Producer tries to Describe topics it has no access to · Issue #722 · IBM/sarama · GitHub

Here, I would like to confirm:

Does CDC use Golang’s Sarama?
If Sarama is used, is conf.Metadata.Full=false not configured?
If neither of the above, what could be causing this issue?

translator_bot · June 22, 2024, 8:16pm

| username: neilshen | Original post link

Yes.

It is not configured.

The relevant code is in the link below. Contributions with PRs to solve this issue are welcome~

github.com

pingcap/tiflow/blob/857e7cfbe74620deb1a46a1f4cdfe383d4977fd1/cdc/sink/mq/producer/kafka/config.go#L336


      
          // DeriveTopicConfig derive a `topicConfig` from the `Config`
          func (c *Config) DeriveTopicConfig() *AutoCreateTopicConfig {
          	return &AutoCreateTopicConfig{
          		AutoCreate:        c.AutoCreate,
          		PartitionNum:      c.PartitionNum,
          		ReplicationFactor: c.ReplicationFactor,
          	}
          }
          
          // NewSaramaConfig return the default config and set the according version and metrics
          func NewSaramaConfig(ctx context.Context, c *Config) (*sarama.Config, error) {
          	config := sarama.NewConfig()
          
          	version, err := sarama.ParseKafkaVersion(c.Version)
          	if err != nil {
          		return nil, cerror.WrapError(cerror.ErrKafkaInvalidVersion, err)
          	}
          	var role string
          	if contextutil.IsOwnerFromCtx(ctx) {
          		role = "owner"
          	} else {

translator_bot · June 22, 2024, 8:16pm

| username: Cloud王 | Original post link

Is there any configuration in cdc that can make the golang conf.Metadata.Full=false configuration take effect?

translator_bot · June 22, 2024, 8:16pm

| username: neilshen | Original post link

There is no such configuration , we have not encountered such a situation before this post.
Currently, we can only adjust the code to add this configuration.

translator_bot · June 22, 2024, 8:16pm

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.