Drainer component fails to start

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: drainer组件无法启动

| username: TiDBer_X63lsc2J

[TiDB Usage Environment]
Learning and Testing
[TiDB Version]
v6.5.2
[Reproduction Path]
Deploy binlog, scale out pump and drainer, drainer fails to start
[Encountered Problem: Symptoms and Impact]
Deploy binlog, scale out pump and drainer, drainer fails to start
[Resource Configuration]


[Attachments: Screenshots/Logs/Monitoring]
Scale-out configuration file:
image
Execute scale-out: drainer fails to start


No error reason in the logs:
No records in drainer_stderr.log
image
/root/.tiup/logs/tiup-cluster-debug-2023-06-08-18-04-45.log has no specific error reason

Cluster status:

| username: Billmay表妹 | Original post link

To deploy Binlog in a TiDB cluster, you need to use the TiDB Binlog components, which include TiDB Binlog Server, Pump, and Drainer. If Drainer fails to start after scaling out Pump and Drainer, it may be due to missing information about the newly scaled-out Pump in Drainer’s configuration file.

Specifically, after scaling out Pump, you need to add the new Pump’s information to Drainer’s configuration file so that Drainer can correctly synchronize data from the new Pump to downstream Kafka or TiDB clusters. You can follow these steps to modify Drainer’s configuration file:

  1. Go to the machine where Drainer is located and find Drainer’s configuration file. By default, Drainer’s configuration file is located in the drainer-<port> directory under the deploy directory of the TiDB cluster, where <port> is Drainer’s listening port.

  2. Open Drainer’s configuration file, usually named drainer.toml, and find the [syncer] section.

  3. In the [syncer] section, find the db-type and dest-db-type parameters and change their values to mysql and tidb, respectively.

  4. In the [syncer] section, find the to parameter and change its value to the address and port number of the newly scaled-out Pump. For example, if you have scaled out a new Pump with the address 192.168.0.100 and port number 8250, you can change the to parameter’s value to:

    to = "mysql://user:password@192.168.0.100:8250"
    

    where user and password are the username and password of the Pump.

  5. Save Drainer’s configuration file and restart Drainer. You can use the following command to restart Drainer:

    tiup cluster restart <cluster-name> -N <drainer-host>:<drainer-port>
    

    where <cluster-name> is the name of the TiUP cluster, and <drainer-host> and <drainer-port> are the IP address and listening port of Drainer, respectively.

  6. Check Drainer’s logs to see if there are any error messages. If Drainer still fails to start, check if Drainer’s configuration file is correct and ensure that the network connection between Drainer and Pump is normal.

| username: TiDBer_X63lsc2J | Original post link

My pump and drainer were scaled together, and both were successfully scaled. However, the drainer alone won’t start. I followed your instructions, but it still doesn’t work. The log situation is still the same as before.

| username: Raymond | Original post link

Has the drainer server been configured with sudo?

| username: TiDBer_X63lsc2J | Original post link

All my nodes are using the same virtual machine, just with different ports, and they are all operated with the root user, so there shouldn’t be any sudo operations. The logs also don’t indicate any permission issues, and the video tutorial doesn’t include such operations.

| username: Raymond | Original post link

  1. Check if there are any errors in this.
    image

  2. Go to the server where drainer is located and start it with systemctl start drainer-8249, then check the operating system logs to see if there are any errors.

| username: TiDBer_X63lsc2J | Original post link

That is to say, no log files are generated in the log directory of drainer, and starting drainer with systemctl also fails.

| username: Raymond | Original post link

Using systemctl to start drainer also fails to start it. Check the operating system logs.

| username: TiDBer_X63lsc2J | Original post link

Operating system log:

drainer.toml file:

Expansion file:
image

It looks like the error is related to the drainer configuration, but I don’t know where the problem is.
My source is tidb: 192.168.40.113, 4000; the target is mysql, 192.168.40.1, 3306

| username: Raymond | Original post link

Refer to the configuration here.

| username: TiDBer_X63lsc2J | Original post link

The automatically generated configuration file drainer.toml after scaling out is indeed incorrect. The following is a version that can start normally, with an additional line [syncer.to] compared to the original.

image

Thank you very much for your guidance!

| username: TiDBer_X63lsc2J | Original post link

However, I just modified the TiDB configuration and reloaded it to take effect. Then the drainer configuration file drainer.toml automatically reverted to its original auto-generated state, causing it to fail to start again. How can I permanently modify the drainer.toml file?

| username: TiDBer_X63lsc2J | Original post link

Got it, my initial expansion file was incorrect, which resulted in the wrong content in the generated drainer.toml file. Now I know how to modify the configuration. Thank you!

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.