TiCDC Create Extraction Task Error: [CDC:ErrVersionIncompatible] version is incompatible

translator_bot · June 23, 2024, 10:29am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC 创建抽取任务报错Error: [CDC:ErrVersionIncompatible]version is incompatible

| username: 麻烦是朋友

【 TiDB Usage Environment】
Test Environment

【 TiDB Version】
$ tiup cluster display calctidb-cluster
Cluster name: calctidb-cluster
Cluster version: v6.0.0
Deploy user: tidb
SSH type: builtin

【 cdc Version】
$ tiup cdc version
tiup is checking updates for component cdc …
Starting component cdc: /home/tidb/.tiup/components/cdc/v6.1.0/cdc version
Release Version: v6.1.0
Git Commit Hash: 9f5e3cea2fae0a3dce6cc5ca79e3bcd654b8aa99
Git Branch: heads/refs/tags/v6.1.0
UTC Build Time: 2022-06-07 11:54:04
Go Version: go version go1.18.2 linux/amd64
Failpoint Build: false

【 OS Version】
CentOS Linux release 7.8.2003 (Core)

【 tiup Version】
[tidb@ptidbcalcmanage01 ~]$ tiup --version
1.10.2 tiup
Go Version: go1.18.3
Git Ref: v1.10.2
GitHash: 2de5b500c9fae6d418fa200ca150b8d5264d6b19

【Encountered Problem】
Creating a TiCDC sync task in TiDB version 5.0.0 results in the following error:
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Upgrading TiDB to version 6.0.0 in offline mode and creating a TiCDC sync task still results in the same error:
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

【Reproduction Path】
1: Upgraded both the TiDB cluster and tiup components when the error occurred.
2: The same error persisted, so I scaled down the CDC node and then scaled it up again. Concerned that TiCDC might have retained information from the TiDB 5.0 version, I reinstalled it.

【Create Task Command】
tiup cdc cli changefeed create --pd=http://xxx.xx.3.115:2379 --sink-uri=“kafka:///xxx.xx.8.24:15502,/xxx.xx.8.25:15502,172.18.8.26:15502/t_pay_detail?kafka-version=3.1.0&max-message-bytes=10485760&replication-factor=3” --changefeed-id=“t-pay-detail” --config /home/tidb/cdc-conf/t_pay_detail.toml

translator_bot · June 23, 2024, 10:29am

| username: TammyLi | Original post link

The CDC version and TiDB version need to be consistent. When deploying TiCDC, you can use the tiup cluster scale-out cdc.yaml method, which will automatically deploy the same version of CDC.

translator_bot · June 23, 2024, 10:29am

| username: wluckdog | Original post link

I have also tried this operation, but the result is still the same.

translator_bot · June 23, 2024, 10:29am

| username: TammyLi | Original post link

Is your environment’s ticdc & tidb versions completely consistent now? If so, if possible, clear the metadata (this will delete all changefeeds): cdc cli unsafe reset

translator_bot · June 23, 2024, 10:29am

| username: 麻烦是朋友 | Original post link

–CDC Version
[tidb@ptidbcalcmanage01 ~]$ tiup cdc version
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc version
Release Version: v6.0.0
Git Commit Hash: 2f363f653a49fc557ebaab1afa7a47f900fbfc8f
Git Branch: heads/refs/tags/v6.0.0
UTC Build Time: 2022-03-31 10:05:34
Go Version: go version go1.18 linux/amd64
Failpoint Build: false
–TiDB Version
tiup cluster list;
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.10.2/tiup-cluster list
Name User Version Path PrivateKey

calctidb-cluster tidb v6.0.0 /home/tidb/.tiup/storage/cluster/clusters/calctidb-cluster /home/tidb/.tiup/storage/cluster/clusters/calctidb-cluster/ssh/id_rsa
–Creating task still reports the same error.
[tidb@ptidbcalcmanage01 cdc-conf]$ tiup cdc cli changefeed create --pd=http://xxx.xx.3.113:2379 --sink-uri=“kafka://xxx.xx.8.24:15502,xxx.xx.8.25:15502,xxx.xx.8.26:15502/t_pay_detail?kafka-version=3.1.0&max-message-bytes=10485760&replication-factor=3” --changefeed-id=“t-pay-detail” --config /home/tidb/cdc-conf/t_pay_detail.toml
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli changefeed create --pd=http://xxx.xx.3.113:2379 --sink-uri=kafka://xxx.xx.8.24:15502,xxx.xx.8.25:15502,xxx.xx.8.26:15502/t_pay_detail?kafka-version=3.1.0&max-message-bytes=10485760&replication-factor=3 --changefeed-id=t-pay-detail --config /home/tidb/cdc-conf/t_pay_detail.toml
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Usage:
cdc cli changefeed create [flags]

Flags:
-c, --changefeed-id string Replication task (changefeed) ID
–config string Path of the configuration file
–cyclic-filter-replica-ids uints (Experimental) Cyclic replication filter replica ID of changefeed (default )
–cyclic-replica-id uint (Experimental) Cyclic replication replica ID of changefeed
–cyclic-sync-ddl (Experimental) Cyclic replication sync DDL of changefeed (default true)
–disable-gc-check Disable GC safe point check
-h, --help help for create
–no-confirm Don’t ask user whether to ignore ineligible table
–opts key=value Extra options, in the key=value format
–sink-uri string sink uri
–sort-engine string sort engine used for data sort (default “unified”)
–start-ts uint Start ts of changefeed
–sync-interval duration (Experimental) Set the interval for syncpoint in replication(default 10min) (default 10m0s)
–sync-point (Experimental) Set and Record syncpoint in replication(default off)
–target-ts uint Target ts of changefeed
–tz string timezone used when checking sink uri (changefeed timezone is determined by cdc server) (default “SYSTEM”)

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

Error without adding --pd parameter
[tidb@ptidbcalcmanage01 ~]$ tiup cdc cli unsafe reset
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli unsafe reset
Confirm that you know what this command will do and use it at your own risk [Y/N]
Y

Error: fail to open PD client, please check pd address “http://127.0.0.1:2379”: context deadline exceeded
Usage:
cdc cli unsafe reset [flags]

Flags:
-h, --help help for reset

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–no-confirm Don’t ask user whether to confirm executing meta command
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

fail to open PD client, please check pd address “http://127.0.0.1:2379”: context deadline exceeded

Error with adding --pd parameter: CDC:ErrVersionIncompatible
[tidb@ptidbcalcmanage01 ~]$ tiup cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
Confirm that you know what this command will do and use it at your own risk [Y/N]
Y
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Usage:
cdc cli unsafe reset [flags]

Flags:
-h, --help help for reset

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–no-confirm Don’t ask user whether to confirm executing meta command
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

translator_bot · June 23, 2024, 10:29am

| username: 麻烦是朋友 | Original post link

After scaling in, the cleanup also encounters the same issue.
tiup cluster scale-in calctidb-cluster --node xxx.xx.3.116:8300
tiup cluster scale-in calctidb-cluster --node xxx.xx.3.117:8300

[tidb@ptidbcalcmanage01 ~]$ tiup cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
Confirm that you know what this command will do and use it at your own risk [Y/N]
Y
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Usage:
cdc cli unsafe reset [flags]

Flags:
-h, --help help for reset

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–no-confirm Don’t ask user whether to confirm executing meta command
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

translator_bot · June 23, 2024, 10:29am

| username: 麻烦是朋友 | Original post link

I have internally reproduced your issue. The cause is related to your cluster being upgraded from version v5.0.0 to v6.0.0. There is a bug in v6.0.0’s PD. The solution is to upgrade the cluster to version v6.1.0. I have verified that upgrading resolves the issue.

Issue Confirmation:
There is version information of 5.0.0 in PD:
curl http://xxx.xx.3.115:2379/pd/api/v1/stores?state=0\&state=1\&state=2
{
“count”: 36,
“stores”: [
{
“store”: {
“id”: 10,
“address”: “xxx.xx.3.76:20172”,
“state”: 2,
“labels”: [
{
“key”: “host”,
“value”: “xxx.xx.3.76” ---------------> previous host IP
}
],
“version”: “5.0.0”,
** “status_address”: “xxx.xx.3.76:20182”,**
“git_hash”: “7706b9634bd901c9fe8dbe6a556025abbfd0793d”,
“start_timestamp”: 1625815970,
“deploy_path”: “/tidb/tikv2/bin”,
“last_heartbeat”: 1631172405972679111,
“node_state”: 3,
“state_name”: “Tombstone”
},
After upgrading the cluster version, this information still exists. Use the following command to clean it up, which cannot be done in version 6.0.0.
tiup ctl:v6.0.0 pd -u xxx.xx.3.115:2379 store remove-tombstone
After re-verifying, there is no 5.0.0 information, and the CDC task can be created normally again.
I hope this post is helpful to everyone.

translator_bot · June 23, 2024, 10:30am

| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.