TiCDC Incremental Synchronization to Kafka Reports Error

translator_bot · June 22, 2024, 4:05am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc增量同步到kafka报错误

| username: Hacker_lBbigVlk

Kafka authentication method without certificate:
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=‘user’ password=‘password’;

Command:
tiup ctl:v6.1.3 cdc changefeed create --pd=http://10.24.7.1:2379 --sink-uri=“kafka://kafka-rshaig1-0.kafka.com:9092,kafka-rshaig1-1.kafka.com:9092,kafka-rshaig1-2.kafka.com:9092/das_news?kafka-version=2.7.1&sasl-user=user&sasl-password=password&sasl-mechanism=SCRAM-SHA-256&partition-num=3&max-message-bytes=10485760&replication-factor=3&protocol=canal-json” --changefeed-id=“changefeed-vv1” --config=/home/tidb/changefeed-vv1.toml

Return information:
Starting component ctl: /home/tidb/.tiup/components/ctl/v6.1.3/ctl cdc changefeed create --pd=http://10.24.7.1:2379 --sink-uri=kafka://kafka-rshaig1-0.kafka.com:9092,kafka-rshaig1-1.kafka.com:9092,kafka-rshaig1-2.kafka.com:9092/das_news?kafka-version=2.7.1&sasl-user=user&sasl-password=password&sasl-mechanism=SCRAM-SHA-256&partition-num=3&max-message-bytes=10485760&replication-factor=3&protocol=canal-json --changefeed-id=changefeed-vv1 --config=/home/tidb/changefeed-vv1.toml
[2023/07/10 11:03:22.674 +08:00] [WARN] [sink.go:167] [“protocol is specified in both sink URI and config filethe value in sink URI will be usedprotocol in sink URI:canal-json, protocol in config file:default”]
[WARN] some tables are not eligible to replicate, model.TableName{model.TableName{Schema:“news_v8”, Table:“news_class”, TableID:0, IsPartition:false}}
Could you agree to ignore those tables, and continue to replicate [Y/N]
Y
[2023/07/10 11:03:24.971 +08:00] [WARN] [sink.go:167] [“protocol is specified in both sink URI and config filethe value in sink URI will be usedprotocol in sink URI:canal-json, protocol in config file:canal-json”]
Error: [CDC:ErrKafkaNewSaramaProducer]new sarama producer: Cluster authorization failed.
Usage:
cdc cli changefeed create [flags]

Flags:
-c, --changefeed-id string Replication task (changefeed) ID
–config string Path of the configuration file
–cyclic-filter-replica-ids uints (Experimental) Cyclic replication filter replica ID of changefeed (default )
–cyclic-replica-id uint (Experimental) Cyclic replication replica ID of changefeed
–cyclic-sync-ddl (Experimental) Cyclic replication sync DDL of changefeed (default true)
–disable-gc-check Disable GC safe point check
-h, --help help for create
–no-confirm Don’t ask user whether to ignore ineligible table
–opts key=value Extra options, in the key=value format
–schema-registry string Avro Schema Registry URI
–sink-uri string sink uri
–sort-engine string sort engine used for data sort (default “unified”)
–start-ts uint Start ts of changefeed
–sync-interval duration (Experimental) Set the interval for syncpoint in replication(default 10min) (default 10m0s)
–sync-point (Experimental) Set and Record syncpoint in replication(default off)
–target-ts uint Target ts of changefeed
–tz string timezone used when checking sink uri (changefeed timezone is determined by cdc server) (default “SYSTEM”)

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrKafkaNewSaramaProducer]new sarama producer: Cluster authorization failed.
Error: exit status 1

Domain name and account permissions have been checked, all are fine. How should the command be modified?

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

It’s best to confirm,

whether it is plaintext or SCRAM-SHA-256.

Looking at your configuration parameters, there doesn’t seem to be any issue, but the authentication still didn’t pass…

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

It’s not plaintext. The provider gave this:
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

Can you connect using other client tools? Or verification tools?

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

Sure, I specifically had someone test it, and it is readable and writable.

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

Sure, please provide the text you need translated.

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

DEBUG does not have useful information.
Where is the [kafka-client] configuration, in cdc.conf?

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

No, in the configuration file, you can specify

github.com

pingcap/docs-cn/blob/master/ticdc/ticdc-changefeed-config.md

---
title: TiCDC Changefeed 命令行参数和配置参数
summary: 了解 TiCDC Changefeed 详细的命令行参数和配置文件定义。
---

# TiCDC Changefeed 命令行参数和配置参数

## TiCDC Changefeed 命令行参数

本章节将以创建同步任务为例，介绍 TiCDC Changefeed 的命令行参数：

```shell
cdc cli changefeed create --server=http://10.0.10.25:8300 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --changefeed-id="simple-replication-task"
```

```shell
Create changefeed successfully!
ID: simple-replication-task
Info: {"upstream_id":7178706266519722477,"namespace":"default","id":"simple-replication-task","sink_uri":"mysql://root:xxxxx@127.0.0.1:4000/?time-zone=","create_time":"2024-05-24T15:05:46.679218+08:00","start_ts":438156275634929669,"engine":"unified","config":{"case_sensitive":false,"enable_old_value":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":3600000000000,"filter":{"rules":["test.*"],"event_filters":null},"mounter":{"worker_num":16},"sink":{"protocol":"","schema_registry":"","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false},"column_selectors":null,"transaction_atomicity":"none","encoder_concurrency":16,"terminator":"\r\n","date_separator":"none","enable_partition_separator":false},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"storage":""}},"state":"normal","creator_version":"v8.1.0"}
```

This file has been truncated. show original

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

Does version 6.1.3 support these parameters?
[sink.kafka-config]
sasl-mechanism = “SCRAM-SHA-256”
sasl-user = “user”
sasl-password = “password”

Error:
component TiCDC changefeed’s config file /home/tidb/changefeed-news-v8.toml contained unknown configuration options: sink.kafka-config, sink.kafka-config.sasl-mechanism, sink.kafka-config.sasl-user, sink.kafka-config.sasl-password

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

6.1 seems not to support it,

It seems that this configuration mode is only supported after 6.5

github.com

pingcap/docs-cn/blob/release-6.5/ticdc/ticdc-changefeed-config.md

---
title: TiCDC Changefeed 命令行参数和配置参数
summary: 了解 TiCDC Changefeed 详细的命令行参数和配置文件定义。
---

# TiCDC Changefeed 命令行参数和配置参数

## TiCDC Changefeed 命令行参数

本章节将以创建同步任务为例，介绍 TiCDC Changefeed 的命令行参数：

```shell
cdc cli changefeed create --server=http://10.0.10.25:8300 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --changefeed-id="simple-replication-task"
```

```shell
Create changefeed successfully!
ID: simple-replication-task
Info: {"upstream_id":7178706266519722477,"namespace":"default","id":"simple-replication-task","sink_uri":"mysql://root:xxxxx@127.0.0.1:4000/?time-zone=","create_time":"2024-06-20T15:05:46.679218+08:00","start_ts":438156275634929669,"engine":"unified","config":{"case_sensitive":false,"enable_old_value":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":3600000000000,"filter":{"rules":["test.*"],"event_filters":null},"mounter":{"worker_num":16},"sink":{"protocol":"","schema_registry":"","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false},"column_selectors":null,"transaction_atomicity":"none","encoder_concurrency":16,"terminator":"\r\n","date_separator":"none","enable_partition_separator":false},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"storage":""}},"state":"normal","creator_version":"v6.5.10"}
```

This file has been truncated. show original

Are you using the command line configuration mode, and it still doesn’t work?

$ kafka-console-producer.sh --broker-list --topic --producer.config --property “sasl.mechanism=SCRAM-SHA-256” --property “sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username= password=;”

Referring to this command is successful, right?

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

Kafka can read and write normally. The target Kafka version is 2.7.1.
Today, I used the higher version Kafka 3.1.2 client with kafka-console-producer.sh for testing and encountered a “Cluster authorization failed” error.
Using the Kafka 2.7.1 client with kafka-console-producer.sh for testing, it can read and write normally.
May I ask if the Kafka client for TiCDC also has version distinctions? TiCDC is v6.1.3; does this version support Kafka 2.7.1?

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

Here is the version description.

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

Our downstream Kafka version is 2.7.1, which is supported. Why are there still authentication issues?

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

Is it possible to use plaintext instead of authentication?

It still feels like the support is not good enough…

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

No authentication is acceptable.

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

Then it is estimated that this version is not supported yet… If you want to use 6.5.x, it is best to use 6.5.2. 6.5.3 has a major bug, ticdc, so be careful with that.

If plaintext works, then just use plaintext with the current version, no need to fuss about it

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

Okay, let’s test version 6.5.2 in our testing environment.

translator_bot · June 22, 2024, 4:05am

| username: Hacker_lBbigVlk | Original post link

It turns out my Kafka account only had WRITE and READ permissions. Later, I added the Create and Describe permissions for the Topic resource type and the DescribeConfigs permission for the Cluster resource type.

The official documentation states:

We didn’t notice this before. The issue is now resolved. Thank you for your support.

translator_bot · June 22, 2024, 4:05am

| username: xfworld | Original post link

Well, I was just looking at the release notes for 6.1.7. A lot of bugs have been fixed, so it might be worth considering.

translator_bot · June 22, 2024, 4:05am

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.