Data Loss During Drainer Synchronization

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: drainer同步时丢失数据

| username: qufudcj

[TiDB Usage Environment] Test
[TiDB Version] v4.0.4
[Reproduction Path]
Run large transaction operations
[Encountered Problem: Phenomenon and Impact]
A batch of data needs to be initialized. Considering that the database involved in this batch of data has already been specified in the drainer’s syncer.ignore-schemas configuration, I temporarily adjusted the tidb configuration performance.txn-total-size-limit from 800M to 8G, but the drainer still had issues.

From the phenomenon, it seems that the drainer lost some data during synchronization, and there were errors connecting to the pump during that period. The pump itself also reported errors. It feels like after 9:56, the drainer only pulled part of the binlog. I tried to consume the data in Kafka myself and found that the data in Kafka was incomplete, specifically reflected in the discontinuity of the SQL IDs of a certain table.

It looks like the drainer can no longer read the binlog from a certain pump? I have three pump instances here.

What configuration of the pump do I need to adjust to avoid this situation?

And how do I repair the data?

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
drainer logs
drainer.log (115.2 KB)
pump logs
pump_stderr.log (9.7 KB)
pump.log (54.3 KB)
SQL missing in downstream database, IDs are not continuous

| username: weixiaobing | Original post link

You can check the monitoring to see if there is a TiDB server that skipped the binlog.

| username: qufudcj | Original post link

Indeed, there is an issue, and it returned to normal after I restarted the tidb-server at noon. How can I trace the cause of this? I want to avoid this situation next time.

| username: weixiaobing | Original post link

Search for the keyword “binlog” in the logs of the TiDB server node where “skip binlog” occurred. It should record the reason for “skip binlog”. Based on your mention of deleting large transactions, it is likely caused by this large transaction deletion. You can use the startts in the logs to find the specific SQL in the slow logs.

| username: qufudcj | Original post link

I see a lot of errors in the tidb-server logs indicating it can’t connect to the pump port.

[2023/06/09 09:34:36.803 +08:00] [INFO] [main.go:332] [tidb-server] ["create pumps client success, ignore binlog error"=true]
[2023/06/09 09:55:44.237 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068575891462] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068615213059] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068431712258] ["commit ts"=0] [length=276] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068405497876] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068130246662] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068707225602] ["commit ts"=0] [length=246] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068366176258] ["commit ts"=0] [length=188] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:45.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:46.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:47.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:48.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:49.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:50.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:51.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:52.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:53.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:54.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:55.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:56.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:57.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:58.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:59.338 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.18.27:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]

Is TiDB actively writing binlogs to the pump? I always thought the pump actively fetched them from the tidb-server.

Currently, I observe that only one pump instance had slightly higher memory usage at that time, but it was far from the host’s memory limit, and the disk space is sufficient. I’m not sure how to proceed with the investigation.

| username: weixiaobing | Original post link

How many nodes does your pump have? Also, check if there are any logs related to binlog ERROR.

| username: Raymond | Original post link

Check your configuration parameter
binlog.ignore-error
Is it possible that TiDB failed to write to the pump and then skipped the pump writing stage?

| username: redgame | Original post link

Looks like it hit something.

| username: qufudcj | Original post link

It is enabled, and according to the documentation, once this error is triggered, binlog writing stops. The current issue seems to be why binlog writing fails. Moving forward, we want to avoid the situation where binlog writing fails.

| username: qufudcj | Original post link

Looking at the logs, it seems like a large transaction caused the pump to malfunction, and then the tidb-server couldn’t connect to the pump, triggering binlog.ignore-error. Since then, this tidb-server has stopped writing binlogs. Because I don’t understand the Go language, I don’t know why this large transaction caused the pump to malfunction.

| username: qufudcj | Original post link

Three nodes, the logs of the three nodes are similar. I have attached them. I didn’t find any binlog-related errors, but there are other errors.

| username: qufudcj | Original post link

In the pump_stderr.log, combined with the “start pump…” in the pump.log, analyze what triggered it to crash and restart.

| username: 像风一样的男子 | Original post link

Have you ever scaled the nodes up or down, or restarted TiDB in the meantime?

| username: qufudcj | Original post link

On that day, there was no issue; we just lifted the transaction size limit and initialized a large batch of data. However, the tables related to the initialized data have already been filtered in the drainer.

| username: 像风一样的男子 | Original post link

Previously, my drainer also experienced data loss, but the reason was that after scaling in and out nodes, the binlog configuration of TiDB was automatically changed to the off state.

| username: qufudcj | Original post link

It should be different from my situation. In my case, synchronization is normal before performing large transactions, and it returns to normal after restarting the tidb-server.

| username: Anna | Original post link

Achieving a consistent state in the downstream cluster means that the data in the downstream cluster is equivalent to a snapshot of the upstream cluster with tidb_snapshot = ts set.

Checkpoint consistency means that the Drainer checkpoint has saved the synchronization consistency state through consistent. When Drainer is running, consistent is false, and after Drainer exits normally, consistent is updated to true.

| username: qufudcj | Original post link

Uh… This doesn’t seem to explain the cause of the pump anomaly, does it?

| username: Raymond | Original post link

Check the corresponding time period in tidb.log to see what information is inside.

| username: qufudcj | Original post link

TiDB also cannot connect to the pump.

[2023/06/09 09:34:36.803 +08:00] [INFO] [main.go:332] [tidb-server] ["create pumps client success, ignore binlog error"=true]
[2023/06/09 09:55:44.237 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068575891462] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068615213059] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068431712258] ["commit ts"=0] [length=276] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068405497876] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068130246662] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068707225602] ["commit ts"=0] [length=246] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068366176258] ["commit ts"=0] [length=188] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:45.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:46.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:47.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:48.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:49.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:50.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:51.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:52.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:53.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:54.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:55.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:56.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:57.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:58.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:59.338 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.18.27:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]