Binlog Master-Slave Delay

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: binlog主备延迟

| username: 答辩潮人

[TiDB Usage Environment] Production Environment
[TiDB Version] v4.0.15
[Reproduction Path]
[Encountered Problem: Phenomenon and Impact]


Using binlog for master-slave synchronization, we found that the commit delay is a bit high. Are there any parameters that can be optimized?
Ignoring version upgrade situations.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: 芮芮是产品 | Original post link

Your latency is still acceptable.

| username: TI表弟 | Original post link

Hahaha, why is the version so low?

| username: 答辩潮人 | Original post link

Binlog monitoring metrics

| username: TiDBer_小阿飞 | Original post link

The synchronization delay of TiDB Binlog is at the second level, generally around 3 seconds during non-peak business hours. Enabling TiDB Binlog has a slight performance impact on transactions that involve writing or updating data. In terms of delay, during the Prewrite phase, a transaction can only be committed after a p-binlog is successfully written concurrently. Generally, writing binlog is faster than KV Prewrite, so it does not increase the delay. You can see the response time for writing binlog on the Pump monitoring panel.

| username: 像风一样的男子 | Original post link

A master-slave delay of a few hundred milliseconds is not considered high.

| username: 答辩潮人 | Original post link

However, when there are many tasks, the latency increases. Currently, the primary-secondary delay is 10 minutes.

| username: TI表弟 | Original post link

Why not enable binlog and set up multiple four-region three-center backups for mutual redundancy?

| username: 答辩潮人 | Original post link

There has been an increase, do I need to adjust the parameters?

| username: 答辩潮人 | Original post link

Normally at 200ms, there is no delay between primary and standby. When it reaches 400ms, the delay starts to increase.

| username: oceanzhang | Original post link

First, check if the latency is still increasing. If it remains within a certain range, you don’t need to worry about it.

| username: TiDBer_小阿飞 | Original post link

  • If there is a large delay between Drainer and the downstream, you can increase the Drainer worker-count parameter (for cross-data center synchronization, it is recommended to deploy Drainer in the downstream).
  • If the downstream load is not high, you can try increasing the Drainer worker-count parameter.
    TiDB Binlog 常见问题 | PingCAP 文档中心
| username: 像风一样的男子 | Original post link

What configuration of pump and drainer can increase concurrency?

| username: 答辩潮人 | Original post link

That kind of architecture

| username: 答辩潮人 | Original post link

txn-batch 50
worker-count 108

| username: TI表弟 | Original post link

There is a need to synchronize a copy to Kafka using TiCDC.

| username: 答辩潮人 | Original post link

It’s precisely because of the increase that adjustments are needed. Now it’s 10 minutes and still going up.

| username: 答辩潮人 | Original post link

Version 4.0.15, version 5.0 has already switched to CDC, and upgrading the version is also quite troublesome. It is still in the planning stage.

| username: TiDBer_小阿飞 | Original post link

Use tiup to scale out drainer for incremental synchronization and reduce latency.

| username: 答辩潮人 | Original post link

Is there any documentation? I am currently using Drainer to synchronize binlogs to the downstream.