TiCDC Port Conflict During Scaling

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc 扩容端口冲突

| username: GreenGuan

I already have a ticdc on host A with the monitor port number 9100. I plan to expand another ticdc on host A for use by another cluster, but I repeatedly get port conflict errors. I have tried changing the port number, but the same issue persists. Has anyone encountered this before? How did you configure it?

global:
  user: tidb
  ssh_port: 22
  ssh_type: builtin
  deploy_dir: /data2/deploy
  data_dir: /data2

monitored:
  node_exporter_port: 9200
  deploy_dir: "x"
  data_dir: "x"
  log_dir: "x"
  blackbox_exporter_port: 9215

cdc_servers:
- host: x
  ssh_port: 22
  port: 8301
  deploy_dir: x
  log_dir: x

- host: x
  ssh_port: 22
  port: 8301
  deploy_dir: x
  log_dir: x

monitoring_servers:
- host: x
  ssh_port: 22
  port: 9300

The error is as follows:

Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.10.1/tiup-cluster /home/tidb/.tiup/components/cluster/v1.10.1/tiup-cluster scale-out x scale-out.yaml
You have one or more of ["global", "monitored", "server_configs"] fields configured in
	the scale out topology, but they will be ignored during the scaling out process.
	If you want to use configs different from the existing cluster, cancel now and
	set them in the specification fields for each host.
Do you want to continue? [y/N]: (default=N) y

+ Detect CPU Arch Name
  - Detecting node x Arch info ... Done
  - Detecting node x Arch info ... Done

+ Detect CPU OS Name
  - Detecting node x OS info ... Done
  - Detecting node x OS info ... Done

Error: Deploy port conflicts to an existing cluster (spec.deploy.port_conflict)

The port you specified in the topology file is:
  Port:      9100
  Component: monitor x

It conflicts to a port in the existing cluster:
  Existing Cluster Name: xxxxx
  Existing Port:         9100
  Existing Component:    monitor x

Please change to use another port or another host.
2
| username: weixiaobing | Original post link

You only need to scale out CDC, the rest does not need to be scaled out.

3
| username: GreenGuan | Original post link

Expanding only TiCDC will also report an error. I noticed that the node_exporter_port port number seems to be set when the cluster is initially established. If the port number for cluster A is 9100 (default) and a set of TiCDC is deployed on host A, then if cluster B wants to expand a new TiCDC node on host A, unless the default port of node_exporter_port is modified (to avoid 9100), there will be a port conflict. There are two awkward points:

  1. It is not possible to configure the node_exporter_port port number for TiCDC separately.
  2. If you want to modify the node_exporter_port port number of the cluster, you need to restart the entire instance (I checked again and it seems that it cannot be modified).
4
| username: neilshen | Original post link

Refer to the following example to scale out TiCDC.

$ cat ./scale-out.yaml
cdc_servers:
  - host: 172.16.5.37
    port: 47915
    deploy_dir: /data/cdc/cluster1/tidb-deploy/cdc-47915
    data_dir: /data/cdc/cluster1/tidb-data/cdc-47915

$ tiup cluster scale-out cdc-cluster1 ./scale-out.yaml

Ensure that the port specified in “port” is not occupied by other programs.

5
| username: HACK | Original post link

Check at the operating system level to see if the port you configured is being used.

6
| username: GreenGuan | Original post link

The port for TiCDC is not occupied, but the port for monitoring TiCDC is occupied, causing TiCDC to fail to scale successfully.

7
| username: yilong | Original post link

What I tested is different from yours. If a new expansion is done, 9100 will replace the previous one. So the expansion can be successful.
What are your tiup and cluster versions?

8
| username: GreenGuan | Original post link

May I ask if there is a TiDB or other TiCDC service on the target host for expansion? Is its monitoring port 9100?

tiup version v1.10.1
tidb version v5.4.0

9
| username: yilong | Original post link
  1. Your version is higher than mine.
  2. Just like your environment, there is already a cluster that includes ticdc. In another cluster, a new ticdc node is added on this host. After expansion, the new ticdc will take over the monitor port.
  3. It might be more reasonable in the new version of tiup. If the monitor port already exists, the expansion should not succeed.
  4. However, whether it is possible to change the monitor port to make the expansion successful needs to be considered as a reasonable requirement. Running multiple components of different clusters on one host can lead to resource contention, and if the host has issues, it affects multiple clusters. Is it more appropriate to have one cluster per host? Or is it more reasonable to use k8s to manage different clusters and allocate resources?
  5. If you must modify the node_exporter_port, you can refer to this post: 如何修改node_exporter_port - #5，来自 wfxxh - TiDB 的问答社区
10
| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.