Can DM limit the traffic when migrating MySQL to TiDB?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: DM迁移MySQL到TiDB可以限制流量吗?

| username: Johnpan

When migrating a MySQL database to TiDB using the DM tool, can rate limiting configurations be added on DM? For example, limiting network traffic or restricting the number of CPU cores used?

【TiDB Usage Environment】Production\Test Environment\POC
【TiDB Version】
【Encountered Problem】
【Reproduction Path】What operations were performed that led to the problem
【Problem Phenomenon and Impact】
【Attachments】

  • Relevant logs, configuration files, Grafana monitoring (https://metricstool.pingcap.com/)
  • TiUP Cluster Display information
  • TiUP Cluster Edit config information
  • TiDB-Overview monitoring
  • Corresponding module’s Grafana monitoring (if any, such as BR, TiDB-binlog, TiCDC, etc.)
  • Corresponding module logs (including logs from 1 hour before and after the issue)

If the question is related to performance optimization or troubleshooting, please download and run the script. Please select all and copy-paste the terminal output results for upload.

| username: Hacker_小峰 | Original post link

You can refer to the official website DM configuration optimization to set and adjust parameters such as rows, chunk-filesize, pool-size, batch, etc., to limit the export and import speed:

DM configuration templates can be referred to:

| username: tidb狂热爱好者 | Original post link

Traffic can be limited.

| username: Johnpan | Original post link

The number of CPU cores is limited by using threads, but which parameter is used to configure the network traffic transmitted between MySQL and TiDB?

| username: Johnpan | Original post link

I looked at the configuration on the official website mentioned by the teacher above, but I still don’t quite understand which parameter limits the network traffic between MySQL and TiDB?
Because it’s a production environment, I don’t want the network card traffic to be fully utilized, as it might affect other services.

| username: Hacker007 | Original post link

Try increasing the max-allowed-packet.

| username: xiaohetao | Original post link

These two parameters do not limit the traffic directly, but they should have an indirect effect on network traffic.

| username: Johnpan | Original post link

This parameter primarily affects the TiDB side. Additionally, this parameter only pertains to the size of the result set involved in the SQL, and it doesn’t have much to do with network throttling.

| username: Johnpan | Original post link

What you mentioned is incremental, which should have an indirect effect, but I mainly want to limit the network transmission traffic during DM full migration to reduce the impact on the business.

| username: Johnpan | Original post link

Currently, you can only set the threads for mydumper, such as setting it to 1 or 2 to reduce network traffic.

| username: xiaohetao | Original post link

Then you try adjusting it.

| username: alfred | Original post link

There should be no direct parameters to limit network traffic here. You can adjust threads and chunk-filesize and monitor to see.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.