[Reproduction Path] What operations were performed that caused the issue
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
tidb drainer synchronizes data to kafka, and the drainer log reports the following error
[2023/03/17 17:05:46.675 +08:00] [FATAL] [kafka.go:223] [“fail to produce message to kafka, please check the state of kafka server”] [error=“kafka: Failed to produce message to topic user-binlog: kafka: error encoding packet: invalid request size (5980520)”] [stack=“github.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:223”]
The error persists after adjusting the following drainer parameters
kafka-max-message-size = 1000
kafka-max-messages = 1
The error is as follows
[2023/03/17 17:16:37.051 +08:00] [FATAL] [kafka.go:223] [“fail to produce message to kafka, please check the state of kafka server”] [error=“kafka: Failed to produce message to topic user-binlog: kafka: error encoding packet: invalid request size (28106)”] [stack=“github.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:223”]
When using the default value, the following error occurs, but in fact, Kafka is functioning normally:
[2023/03/17 18:35:56.416 +08:00] [ERROR] [syncer.go:488] [“Failed to close syncer”] [error=“fail to push msg to kafka after 30s, check if kafka is up and working”] [errorVerbose=“fail to push msg to kafka after 30s, check if kafka is up and working\ngithub.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:236\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”]
[2023/03/17 18:36:06.417 +08:00] [ERROR] [util.go:62] [“Recovered from panic”] [err=“"Waiting too long for Syncer.run to quit."”] [“real stack”=“github.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:64\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:965\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:494\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).Start\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:133\ngithub.com/pingcap/tidb-binlog/drainer.(*Server).Start.func4\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/server.go:290\ngithub.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:72”] [name=syncer]
If kafka-max-message-size is changed to 50m, the following error occurs:
[2023/03/18 18:22:09.974 +08:00] [FATAL] [kafka.go:223] [“fail to produce message to kafka, please check the state of kafka server”] [error=“kafka: Failed to produce message to topic user-binlog: kafka server: Unexpected (unknown?) server error.”] [stack=“github.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:223”]
|
username: TiDBer_KkruFifg
|
username: 
|
username: