Drainer process is running, but the port cannot be started

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: drainer进程在，端口无法启动

| username: 普罗米修斯

[TiDB Usage Environment] Production Environment
[TiDB Version]
Upstream TiDB Version: v3.0.3
Downstream TiDB Version: v5.2.4
[Overview] Scenario + Problem Overview
Incremental data from the upstream TiDB is synchronized to the downstream higher version cluster through the drainer service. The drainer service is deployed via ansible, but it fails to start. Checking the process with ps shows it is running, but the port is not opened.

Image
Image1707×117 7.92 KB

Image
Image1696×362 31.8 KB

Image
Image1380×272 45.8 KB

[Resource Configuration]
Drainer server: 203G memory; 10Gbps network interface; 32 cores;
All pump statuses are online:
Image
Image1380×124 35.7 KB

Binlog status is enabled:
Image
Memory is not fully utilized:
Image
Image1380×192 29 KB

Bandwidth situation:
Image
Image1380×360 51.2 KB

2
| username: songxuecheng | Original post link

It looks fine. Suggest adjusting the wait time and restarting.

3
| username: WalterWj | Original post link

Wait a bit, when drainer starts, it will traverse the entire DDL history. If there are many DDLs in this cluster… it might take a very long time.

4
| username: 普罗米修斯 | Original post link

However, this ansible-playbook will report an error and stop.

Image
Image1380×272 45.8 KB

5
| username: WalterWj | Original post link

As this brother said, because Ansible waits for a period of time by default, if it doesn’t start successfully within that time, it will report an error and exit. In reality, it is still in the process of starting up.

6
| username: 普罗米修斯 | Original post link

Okay, then I will continue to monitor the drainer.log log.

7
| username: 普罗米修斯 | Original post link

No, after the memory usage goes up, OOM kills the drainer.

8
| username: songxuecheng | Original post link

Upgrade the memory. In cases with many DDL operations, a lot of memory is needed.

9
| username: 普罗米修斯 | Original post link

Okay, I’ll think about how to do it.

10
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.