After adding new configuration parameters to the CDC in the TiDB cluster, the CDC component fails to start and reports an error: unknown configuration options in conf/cdc.toml - sorter.cache-size-in-mb, sorter.sorter-dir

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb集群中的cdc添加新配置参数后,无法正常启动cdc组件,报错:component TiCDC server’s config file conf/cdc.toml contained unknown configuration options: sorter.cache-size-in-mb, sorter.sorter-dir

| username: vcdog

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.0
[Reproduction Path] Add new configuration parameters to the cdc in the tidb cluster using

# tiup cluster edit-config cluster_name

image

[Encountered Problem: Problem Phenomenon and Impact] After adding new configuration parameters to the cdc in the tidb cluster, the cdc component cannot start normally, and an error is reported:

component TiCDC server's config file conf/cdc.toml contained unknown configuration options: sorter.cache-size-in-mb, sorter.sorter-dir

Check the configuration file of the cdc-server, and the parameters have been loaded normally:

Questions:

  1. After normally modifying and updating the configuration parameters, the cdc component shows normal when starting, but in fact, the cdc service on the backend cdc server did not start successfully.
  2. The version of the cdc in the tidb cluster is also: v6.5.0, and the official cdc_server configuration does indeed have these two parameters:
    docs-cn/ticdc/ticdc-server-config.md at release-6.5 · pingcap/docs-cn · GitHub

[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]

| username: WalterWj | Original post link

The error you encountered is because cdc does not recognize the parameter.

 tiup cdc version
Starting component cdc: /root/.tiup/components/cdc/v8.1.0/cdc version
Release Version: v8.1.0
Git Commit Hash: 137ed650cc3fcd04ed31b4e7978dccebc087330d
Git Branch: HEAD
UTC Build Time: 2024-05-20 13:05:21
Go Version: go version go1.21.10 linux/amd64
Failpoint Build: false

Use a similar syntax to check the version of cdc in the corresponding deployment directory.
| username: dba-kit | Original post link

To add variables in tiup, many need to be placed under config. You can refer to the specific details at 通过 TiUP 部署 TiDB 集群的拓扑文件配置 | PingCAP 文档中心

| username: dba-kit | Original post link

Well, I tried version 7.5, and this method does work. It also reloads successfully.

| username: vcdog | Original post link

On the control machine, execute the command to check the CDC version information as follows:

[root@wbjdnchzttdb01 bj_sjzt_to_slave]# tiup cdc version
tiup is checking updates for component cdc ...
Starting component `cdc`: /root/.tiup/components/cdc/v6.5.0/cdc version
Release Version: v6.5.0
Git Commit Hash: 9e91cff866d240ab6c1737680c17f5c5d0586911
Git Branch: heads/refs/tags/v6.5.0
UTC Build Time: 2022-12-23 08:43:57
Go Version: go version go1.19.3 linux/amd64
Failpoint Build: false

On the remote CDC server component’s server, check the CDC version as follows:

[root@wtj7vpztnctcd01 cdc-8300]# ./bin/cdc version
Release Version: v6.5.0
Git Commit Hash: 9e91cff866d240ab6c1737680c17f5c5d0586911
Git Branch: heads/refs/tags/v6.5.0
UTC Build Time: 2022-12-23 08:43:57
Go Version: go version go1.19.3 linux/amd64
Failpoint Build: false
| username: vcdog | Original post link

I didn’t quite understand the modification suggestion in this passage. Could you provide an example of a configuration change? Thanks.

| username: TIDB-Learner | Original post link

Some versions do not support certain parameter configuration methods.

| username: WalterWj | Original post link

It looks like version 650 does not support this parameter. It seems the documentation was written too early…

| username: dba-kit | Original post link

You can ignore that part. I tried it according to your method, and version 7.5.1 can correctly recognize and reload. So it’s still a version issue, possibly a new feature introduced in a minor version of 6.5. 6.5.0 doesn’t have it.

| username: dba-kit | Original post link

After checking the time when the configuration was added, it seems that the variable was introduced in version 6.5.3, so you need to upgrade the version to use it.
PS: While flipping through the release notes, I found that before version 6.5.4, it seems that each version of CDC introduced some new configuration items, with significant performance improvements. Therefore, when using it, you indeed need to pay attention to the version.

| username: vcdog | Original post link

Okay, it seems that the only remaining option is to upgrade the version.