After upgrading from 6.1.7 to 7.5.0, cdc changefeed creation failed

translator_bot · June 21, 2024, 9:24am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 6.1.7升级7.5.0 后 cdc changefeed 创建失败

| username: Hacker_vdm1U4HH

[TiDB Usage Environment]
Testing Environment

[TiDB Version]
6.17 → 7.5.0

[Reproduction Path]
tiup cdc cli changefeed create --server http://192.168.1.235:8300 --sink-uri=“kafka://192.168.1.250:32201,192.168.1.250:32202,192.168.1.250:32203/cdc_member_preprod?protocol=canal-json&kafka-version=2.5.0&partition-num=6&max-message-bytes=10485760&replication-factor=1” --changefeed-id=“preprod-member-001” --config=changefeed-preprod-member.toml

[Encountered Issue: Problem Phenomenon and Impact]
tiup is checking updates for component cdc …
Starting component cdc: /root/.tiup/components/cdc/v7.5.0/cdc cli changefeed create --server http://192.168.1.235:8300 --sink-uri=kafka://192.168.1.250:32201,192.168.1.250:32202,192.168.1.250:32203/cdc_member_preprod?protocol=canal-json&kafka-version=2.5.0&partition-num=6&max-message-bytes=10485760&replication-factor=1 --changefeed-id=preprod-member-001 --config=changefeed-preprod-member.toml
Error: [CDC:ErrDispatcherFailed]index not found when verify the table, table: preprod_member.mb_member, index:

[Resource Configuration]
cat changefeed-preprod-member.toml

case-sensitive = true
force-replicate = true

[filter]
rules = [‘preprod_member.mb_member’]

[mounter]
worker-num = 8

[sink]
dispatchers = [
{matcher = [‘preprod_member.mb_'], dispatcher = “rowid”},
{matcher = ['preprod_member.wx_’], dispatcher = “rowid”}
]
protocol = “canal-json”

[Attachment: Screenshot/Log/Monitoring]

translator_bot · June 21, 2024, 9:24am

| username: 像风一样的男子 | Original post link

Is this the reason?

translator_bot · June 21, 2024, 9:24am

| username: Hacker_vdm1U4HH | Original post link

This table has a primary key…

translator_bot · June 21, 2024, 9:24am

| username: 像风一样的男子 | Original post link

force-replicate = true This enables synchronization of tables without effective indexes.
Try deleting this parameter.

translator_bot · June 21, 2024, 9:24am

| username: Hacker_vdm1U4HH | Original post link

The problem persists.

translator_bot · June 21, 2024, 9:24am

| username: 小龙虾爱大龙虾 | Original post link

In higher versions, the syntax has changed. It’s no longer the dispatcher of 6.1, but has become partition. Take a look:

[sink]
dispatchers = [
  {matcher = ['test1.*', 'test2.*'], topic = "Topic Expression 1", partition = "ts" },
  {matcher = ['test3.*', 'test4.*'], topic = "Topic Expression 2", partition = "index-value" },
  {matcher = ['test1.*', 'test5.*'], topic = "Topic Expression 3", partition = "table"},
  {matcher = ['test6.*'], partition = "ts"}
]

translator_bot · June 21, 2024, 9:24am

| username: Hacker_vdm1U4HH | Original post link

Change to the following content:
dispatchers = [
{matcher = [‘preprod_member.mb_'], partition = “index-value”, index_name = “uid”},
{matcher = ['preprod_member.wx_’], partition = “index-value”, index_name = “id”}
]

Error:
Error: component TiCDC changefeed’s config file changefeed-preprod-member.toml contained unknown configuration options: sink.dispatchers.index-name, sink.dispatchers.index-name

translator_bot · June 21, 2024, 9:24am

| username: Hacker_vdm1U4HH | Original post link

Currently, setting the partition to the default table can be created successfully.
Could it be that index-value doesn’t work?

translator_bot · June 21, 2024, 9:24am

| username: 小龙虾爱大龙虾 | Original post link

You encountered the error “unknown configuration options: sink.dispatchers.index-name,” which clearly indicates that the configuration is not recognized.

Refer to this documentation: