An error occurred when adding a downstream for CDC

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 为cdc增加下游的时候出现下面的错误

| username: TimeKiller

[TiDB Environment] centos7
[TiDB Version] v5.1.0
[Reproduction Path] The following error occurs when adding a downstream for CDC
[Encountered Problem: Phenomenon and Impact]

Using

./cdc cli changefeed create --pd http://10.206.0.7:2383 --sink-uri mysql://root:123456@127.0.0.1:3306/

to add a downstream for CDC, but an error is displayed when checking the list. The upstream is TiDB v5.1.0, and the downstream is 5.7.43 MySQL Community Server.

| username: Billmay表妹 | Original post link

Synchronization between TiDB instances can be done using TiCDC.

| username: Billmay表妹 | Original post link

For the choice of synchronization tools, check this 同步工具的选择 - #5,来自 小龙虾爱大龙虾 - TiDB 的问答社区

| username: ShawnYan | Original post link

Could you please copy the error message text version?
Are the local and global configurations, as well as the configuration file parameters, all correct?

| username: TimeKiller | Original post link

Yes, I checked it. My synchronization is from TiDB to MySQL, and the tool I chose is TiCDC, which should work. But I don’t know why it is failing.

| username: TimeKiller | Original post link

The error in tmp.log is like this, I couldn’t find that LoadGlobalConfig.

| username: TimeKiller | Original post link

I found that when I created the upstream, the following error already appeared in tmp.log :dizzy_face:

[2023/12/27 13:55:11.898 +08:00] [INFO] [helper.go:54] ["init log"] [file=/tmp/cdc_data/tmp.log] [level=info]
[2023/12/27 13:55:11.898 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v6.5.0-master] [git-hash=82446cd9bd8a2feb52a7a3f403039620aaec9618] [git-branch=master] [utc-build-time="2023-08-12 17:20:07"] [go-version="go version go1.20.6 linux/amd64"] [failpoint-build=false]
[2023/12/27 13:55:11.898 +08:00] [INFO] [helper.go:99] ["using proxy config"] [http_proxy=127.0.0.1:7890] [https_proxy=127.0.0.1:7890] [no_proxy="localhost, 127.0.0.1"]
[2023/12/27 13:55:11.898 +08:00] [INFO] [server.go:130] ["CDC server created"] [pd="[http://10.206.0.7:2383]"] [config="{\"addr\":\"127.0.0.1:8300\",\"advertise-addr\":\"127.0.0.1:8300\",\"log-file\":\"/tmp/cdc_data/tmp.log\",\"log-level\":\"info\",\"log\":{\"file\":{\"max-size\":300000,\"max-days\":0,\"max-backups\":0},\"error-output\":\"stderr\"},\"data-dir\":\"/tmp/cdc_data\",\"gc-ttl\":86400,\"tz\":\"System\",\"capture-session-ttl\":10,\"owner-flush-interval\":50000000,\"processor-flush-interval\":50000000,\"sorter\":{\"num-concurrent-worker\":4,\"chunk-size-limit\":134217728,\"max-memory-percentage\":10,\"max-memory-consumption\":17179869184,\"num-workerpool-goroutine\":16,\"sort-dir\":\"/tmp/sorter\"},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"per-table-memory-quota\":10485760,\"kv-client\":{\"worker-concurrent\":8,\"worker-pool-size\":0,\"region-scan-limit\":40,\"region-retry-duration\":60000000000},\"debug\":{\"table-actor\":{\"event-batch-size\":32},\"enable-pull-based-sink\":true,\"enable-db-sorter\":true,\"db\":{\"count\":8,\"concurrency\":128,\"max-open-files\":10000,\"block-size\":65536,\"writer-buffer-size\":8388608,\"compression\":\"snappy\",\"write-l0-pause-trigger\":2147483647,\"compaction-l0-trigger\":160,\"compaction-deletion-threshold\":10485760,\"compaction-period\":1800,\"iterator-max-alive-duration\":10000,\"iterator-slow-read-duration\":256},\"enable-new-scheduler\":true,\"messages\":{\"client-max-batch-interval\":10000000,\"client-max-batch-size\":8388608,\"client-max-batch-count\":128,\"client-retry-rate-limit\":1,\"server-max-pending-message-count\":102400,\"server-ack-interval\":100000000,\"server-worker-pool-size\":4,\"max-recv-msg-size\":268435456},\"scheduler\":{\"heartbeat-tick\":2,\"collect-stats-tick\":200,\"max-task-concurrency\":10,\"check-balance-interval\":60000000000,\"add-table-batch-size\":50,\"region-per-span\":0},\"enable-new-sink\":true},\"cluster-id\":\"default\"}"]
[2023/12/27 13:55:11.900 +08:00] [INFO] [server.go:457] ["/tmp/cdc_data is set as data-dir (80GB available), sort-dir=/tmp/cdc_data/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2023/12/27 13:55:11.901 +08:00] [INFO] [server.go:300] ["http server is running"] [addr=127.0.0.1:8300]
[2023/12/27 13:55:11.901 +08:00] [INFO] [client.go:405] ["[pd] create pd client with endpoints"] [pd-address="[http://10.206.0.7:2383]"]
[2023/12/27 13:55:11.903 +08:00] [INFO] [base_client.go:360] ["[pd] update member urls"] [old-urls="[http://10.206.0.7:2383]"] [new-urls="[http://10.206.0.12:2383,http://10.206.0.14:2383,http://10.206.0.7:2383]"]
[2023/12/27 13:55:11.903 +08:00] [INFO] [base_client.go:378] ["[pd] switch leader"] [new-leader=http://10.206.0.7:2383] [old-leader=]
[2023/12/27 13:55:11.903 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7316941140801532415]
[2023/12/27 13:55:11.905 +08:00] [INFO] [client.go:698] ["[pd] tso dispatcher created"] [dc-location=global]
[2023/12/27 13:55:11.906 +08:00] [INFO] [client.go:405] ["[pd] create pd client with endpoints"] [pd-address="[http://10.206.0.7:2383]"]
[2023/12/27 13:55:11.908 +08:00] [INFO] [base_client.go:360] ["[pd] update member urls"] [old-urls="[http://10.206.0.7:2383]"] [new-urls="[http://10.206.0.12:2383,http://10.206.0.14:2383,http://10.206.0.7:2383]"]
[2023/12/27 13:55:11.908 +08:00] [INFO] [base_client.go:378] ["[pd] switch leader"] [new-leader=http://10.206.0.7:2383] [old-leader=]
[2023/12/27 13:55:11.908 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7316941140801532415]
[2023/12/27 13:55:11.908 +08:00] [INFO] [client.go:698] ["[pd] tso dispatcher created"] [dc-location=global]
[2023/12/27 13:55:11.909 +08:00] [INFO] [client.go:405] ["[pd] create pd client with endpoints"] [pd-address="[10.206.0.7:2383]"]
[2023/12/27 13:55:11.911 +08:00] [INFO] [base_client.go:360] ["[pd] update member urls"] [old-urls="[http://10.206.0.7:2383]"] [new-urls="[http://10.206.0.12:2383,http://10.206.0.14:2383,http://10.206.0.7:2383]"]
[2023/12/27 13:55:11.911 +08:00] [INFO] [base_client.go:378] ["[pd] switch leader"] [new-leader=http://10.206.0.7:2383] [old-leader=]
[2023/12/27 13:55:11.911 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7316941140801532415]
[2023/12/27 13:55:11.911 +08:00] [INFO] [client.go:698] ["[pd] tso dispatcher created"] [dc-location=global]
[2023/12/27 13:55:11.913 +08:00] [ERROR] [api_client.go:119] ["Fail to add meta region label to PD"] [error="[404] 404 page not found\n"] [errorVerbose="[404] 404 page not found\n\ngithub.com/pingcap/tiflow/pkg/httputil.(*Client).DoRequest\n\tgithub.com/pingcap/tiflow/pkg/httputil/httputil.go:122\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).patchMetaLabel\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:180\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).UpdateMetaLabel.func1\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:117\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).UpdateMetaLabel\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:113\ngithub.com/pingcap/tiflow/pkg/upstream.initUpstream\n\tgithub.com/pingcap/tiflow/pkg/upstream/upstream.go:191\ngithub.com/pingcap/tiflow/pkg/upstream.(*Manager).AddDefaultUpstream\n\tgithub.com/pingcap/tiflow/pkg/upstream/manager.go:92\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).reset\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:237\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).Run\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:297\ngithub.com/pingcap/tiflow/cdc/server.(*server).run.func1\n\tgithub.com/pingcap/tiflow/cdc/server/server.go:356\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/12/27 13:55:11.928 +08:00] [ERROR] [api_client.go:119] ["Fail to add meta region label to PD"] [error="[404] 404 page not found\n"] [errorVerbose="[404] 404 page not found\n\ngithub.com/pingcap/tiflow/pkg/httputil.(*Client).DoRequest\n\tgithub.com/pingcap/tiflow/pkg/httputil/httputil.go:122\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).patchMetaLabel\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:180\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).UpdateMetaLabel.func1\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:117\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).UpdateMetaLabel\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:113\ngithub.com/pingcap/tiflow/pkg/upstream.initUpstream\n\tgithub.com/pingcap/tiflow/pkg/upstream/upstream.go:191\ngithub.com/pingcap/tiflow/pkg/upstream.(*Manager).AddDefaultUpstream\n\tgithub.com/pingcap/tiflow/pkg/upstream/manager.go:92\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).reset\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:237\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).Run\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:297\ngithub.com/pingcap/tiflow/cdc/server.(*server).run.func1\n\tgithub.com/pingcap/tiflow/cdc/server/server.go:356\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/12/27 13:55:11.995 +08:00] [ERROR] [api_client.go:119] ["Fail to add meta region label to PD"] [error="[404] 404 page not found\n"] [errorVerbose="[404] 404 page not found\n\ngithub.com/pingcap/tiflow/pkg/httputil.(*Client).DoRequest\n\tgithub.com/pingcap/tiflow/pkg/httputil/httputil.go:122\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).patchMetaLabel\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:180\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).UpdateMetaLabel.func1\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:117\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/pdutil.(*pdAPIClient).UpdateMetaLabel\n\tgithub.com/pingcap/tiflow/pkg/pdutil/api_client.go:113\ngithub.com/pingcap/tiflow/pkg/upstream.initUpstream\n\tgithub.com/pingcap/tiflow/pkg/upstream/upstream.go:191\ngithub.com/pingcap/tiflow/pkg/upstream.(*Manager).AddDefaultUpstream\n\tgithub.com/pingcap/tiflow/pkg/upstream/manager.go:92\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).reset\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:237\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).Run\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:297\ngithub.com/pingcap/tiflow/cdc/server.(*server).run.func1\n\tgithub.com/pingcap/tiflow/cdc/server/server.go:356\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/12/27 13:55:11.995 +08:00] [WARN] [upstream.go:193] ["Fail to verify region label rule"] [error="[CDC:ErrReachMaxTry]reach maximum try: 3, error: [404] 404 page not found\n: [404] 404 page not found\n"] [upstreamID=7316941140801532415] [upstreamEndpoints="[http://10.206.0.7:2383]"]
[2023/12/27 13:55:11.995 +08:00] [INFO] [upstream.go:210] ["upstream initialize successfully"] [upstreamID=7316941140801532415]
[2023/12/27 13:55:11.995 +08:00] [INFO] [manager.go:98] ["default upstream is added"] [id=7316941140801532415]
| username: 江湖故人 | Original post link

Is the port wrong? I see others using 2379:
Column - TiDB TiCDC Usage Practice | TiDB Community

| username: TimeKiller | Original post link

No, the port I set during deployment is 2383. :thinking:

| username: ShawnYan | Original post link

Is this an internal network environment? Did you enable a proxy?

| username: TimeKiller | Original post link

Hello, I used curl to check, and it can connect to the external network.

| username: xfworld | Original post link

It is necessary to ensure that the network and ports within the node can be directly connected. If there is forwarding, there may be some issues.

| username: dba远航 | Original post link

Check if PD is functioning properly.

| username: Kongdom | Original post link

Looking at the display, it seems that TiCDC is used independently. Which version of TiCDC is being used? Is it compatible with v5.1.0?

| username: Kongdom | Original post link

The display results show that PD is normal. :yum:

| username: zhanggame1 | Original post link

Why is the address 127 here?

| username: Kongdom | Original post link

Judging by the execution command, it should be synchronizing from the TiDB cluster to the local MySQL database, and CDC should also be local.

| username: zhanggame1 | Original post link

The screenshot posted by the original poster is not a single-node deployment either, not sure what’s going on.

| username: Kongdom | Original post link

:thinking: This has nothing to do with whether it is a single-machine deployment. Judging from the title of the original poster, it should be about synchronizing data from the teaching environment to the local environment.

| username: TIDB-Learner | Original post link

It is basically a network connectivity issue. You can test the connectivity between the client and server ports using (telnet ip port).