Migration of Drainer with Kafka as Downstream

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 下游是kafka的drainer迁移

| username: TiDBer_yUoxD0vR

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 3.0.13
[Reproduction Path] What operations were performed to encounter the issue
[Encountered Issue: Issue Phenomenon and Impact]
Now we need to migrate the downstream Kafka drainer to a new machine. Is the following process correct?

  1. Stop the old drainer and record the commitTS in the data file
    cat /tidb/deploy/data.drainer/savepoint
    consistent = false
    commitTS = 445037475525820418
  2. Deploy the new drainer, set the initial_commit_ts in the inventory.ini of tidb-ansible to the above value, and deploy and start the new drainer

The documentation says that Kafka’s checkpoint is saved in a file in the corresponding configuration directory. Is it the value in the old drainer data file data.drainer/savepoint?

[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]

| username: Billmay表妹 | Original post link

The version you’re using seems a bit outdated~ Would you consider upgrading it?

| username: Billmay表妹 | Original post link

Your understanding should be correct.
Here is the process for migrating the downstream Drainer to Kafka:

  1. First, stop the old Drainer process. Before stopping, you need to record the commitTS value from the data file. You can get the commitTS value by checking the /tidb/deploy/data.drainer/savepoint file. The content of the file is similar to:

    consistent = false
    commitTS = 445037475525820418
  2. Next, you need to deploy and start the new Drainer. In the inventory.ini file of TiDB Ansible, set the initial_commit_ts parameter to the commitTS value recorded in the previous step. For example:

    initial_commit_ts = "445037475525820418"
  3. Deploy and start the new Drainer.

During this process, you need to ensure that the old Drainer process has been stopped and the correct commitTS value has been recorded. Then, use that commitTS value to configure the new Drainer and start the new Drainer process.

Regarding Kafka’s checkpoint being saved in the corresponding configuration directory file, your understanding is also correct. The commitTS value is saved in the data.drainer/savepoint file of the old Drainer data file.

Please note that the above process is based on the information you provided and is speculative. The specific operation may vary depending on the environment and configuration. It is recommended to carefully read the official documentation of TiDB and Drainer and operate according to the actual situation before performing the operation.