After Drainer Expansion, No Messages Are Written to Kafka

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: drainer扩容后不往kafka写消息

| username: qufudcj

[TiDB Usage Environment] Test
[TiDB Version] v4.0.4
[Reproduction Path] What operations were performed to encounter the issue
Expanded a new drainer when one drainer had insufficient memory
[Encountered Issue: Problem Phenomenon and Impact]
Both drainers are currently starting normally, but they are not writing messages to Kafka. The Kafka queue exists, and other queues are sending and receiving messages normally. No errors about large messages were found in Kafka.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
New drainer startup log

[2023/04/06 16:53:13.774 +08:00] [INFO] [version.go:50] ["Welcome to Drainer"] ["Release Version"=v4.0.3-dirty] ["Git Commit Hash"=fb0cc0d5290202bb6d1b6f189549f629cfd810ef] ["Build TS"="2021-10-21 02:40:08"] ["Go Version"=go1.15.6] ["Go OS/Arch"=linux/amd64]
[2023/04/06 16:53:13.774 +08:00] [INFO] [main.go:46] ["start drainer..."] [config="{\"log-level\":\"info\",\"node-id\":\"172.16.12.169:8249\",\"addr\":\"http://172.16.12.169:8249\",\"advertise-addr\":\"http://172.16.12.169:8249\",\"data-dir\":\"/alidata/tidb/deploy/data\",\"detect-interval\":5,\"pd-urls\":\"http://172.16.12.73:2379,http://172.16.12.75:2379,http://172.16.12.76:2379\",\"log-file\":\"/alidata/tidb/deploy/log/drainer.log\",\"initial-commit-ts\":0,\"sycner\":{\"sql-mode\":null,\"ignore-txn-commit-ts\":null,\"ignore-schemas\":\"INFORMATION_SCHEMA,METRICS_SCHEMA,PERFORMANCE_SCHEMA,aaa,aliprod119,bak_marketing_0925,dusto_job,goinception,mysql,stress_testing,test,test33,test55,thisaliprod,thisisuat,tidb_loader,txmanager,uat_databank,uat_goods,uat_grid,uat_info,uat_integration,uat_job,uat_marketing,uat_member,uat_message,uat_nacos,uat_order,uat_pay,uat_product,uat_report,uat_saga,uat_security,uat_stock,uat_support,uat_zipkin,pre_databank,pre_info,pre_job,pre_message,pre_nacos,pre_nacos113,pre_product,pre_report,pre_security,pre_support,dev_member,dev_databank,dev_message,dev_info,dev_order,dev_pay,dev_marketing,dev_report,dev_stock,dev_grid,dev_support,dev_security,dev_goods,dev_dusto_dmp,uat_dusto_dmp,pre_dusto_dmp,nacos_dusto_dmp,uat_plan\",\"ignore-table\":[{\"db-name\":\"test\",\"tbl-name\":\"test\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_comp_region_rel_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_category_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_color_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_price_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_size_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_sku_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_clerk_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_data_err\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_store_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"bonus_rate_rule\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"resend_sap_intf\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"send_sap_intf\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"order_pay_mode_bs\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"order_sub_bs\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_detail_provisional\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_range\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_allocation\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_sap_inv_sync_detail\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_sap_inv_sync\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_shelf\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_stock_his\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_stock_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_warn_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_transfer_order\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_transfer_order_detail\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"qrtz_fired_triggers\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"sales_orders\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stock_temp\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"storeinfo_temp\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stores\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"t_post_id\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_detail_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stock_store\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_0611\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_0618\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_2019017\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_bak_0918\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_temp20190515\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_temp20190521\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_transaction_detail_2019017\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_transaction_detail_bak_0918\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp201905_01\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp201905_old\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190521\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190522\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190528\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190529\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190529_01\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_201906011\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190603\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190618\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190624\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190625\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190627\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190701\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190708\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190722\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190729\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190805\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190812\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190820\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190826\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190827\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190912\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190925\"}],\"txn-batch\":20,\"loopback-control\":false,\"sync-ddl\":true,\"channel-id\":0,\"worker-count\":1,\"to\":{\"host\":\"\",\"user\":\"\",\"password\":\"\",\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null},\"encrypted_password\":\"\",\"sync-mode\":0,\"port\":0,\"checkpoint\":{\"type\":\"\",\"schema\":\"\",\"host\":\"\",\"user\":\"\",\"password\":\"\",\"encrypted_password\":\"\",\"port\":0,\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null}},\"dir\":\"\",\"retention-time\":0,\"merge\":false,\"zookeeper-addrs\":\"\",\"kafka-addrs\":\"172.16.16.221:9092,172.16.17.65:9092,172.16.18.27:9092\",\"kafka-version\":\"1.0.2\",\"kafka-max-messages\":1536,\"kafka-client-id\":\"\",\"kafka-max-message-size\":1610612736,\"topic-name\":\"tidb-binlog-pre\"},\"replicate-do-table\":null,\"replicate-do-db\":null,\"db-type\":\"kafka\",\"relay\":{\"log-dir\":\"\",\"max-file-size\":10485760},\"disable-dispatch-flag\":null,\"enable-dispatch-flag\":null,\"disable-dispatch\":null,\"enable-dispatch\":null,\"safe-mode\":false,\"disable-detect-flag\":null,\"enable-detect-flag\":null,\"disable-detect\":null,\"enable-detect\":null},\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null},\"synced-check-time\":5,\"compressor\":\"\",\"EtcdTimeout\":5000000000,\"MetricsAddr\":\"\",\"MetricsInterval\":15}"]
[2023/04/06 16:53:13.775 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[http://172.16.12.73:2379,http://172.16.12.75:2379,http://172.16.12.76:2379]"]
[2023/04/06 16:53:13.778 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:13.778 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:13.778 +08:00] [INFO] [server.go:120] ["get cluster id from pd"] [id=6667489703447796057]
[2023/04/06 16:53:13.780 +08:00] [INFO] [checkpoint.go:64] ["initialize checkpoint"] [type=file] [checkpoint=440603977927622705] [cfg="{\"CheckpointType\":\"file\",\"Db\":null,\"Schema\":\"\",\"Table\":\"\",\"ClusterID\":6667489703447796057,\"InitialCommitTS\":0,\"dir\":\"/alidata/tidb/deploy/data/savepoint\"}"]
[2023/04/06 16:53:13.780 +08:00] [INFO] [store.go:68] ["new store"] [path="tikv://172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379?disableGC=true"]
[2023/04/06 16:53:13.780 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379]"]
[2023/04/06 16:53:13.782 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:13.782 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:13.783 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [client.go:127] ["[sarama] Initializing new client"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [client.go:174] ["[sarama] Successfully initialized new client"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [store.go:68] ["new store"] [path="tikv://172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379?disableGC=true"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379]"]
[2023/04/06 16:53:26.024 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:26.024 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:26.025 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2023/04/06 16:53:26.028 +08:00] [INFO] [server.go:265] ["register success"] ["drainer node id"=172.16.12.169:8249]
[2023/04/06 16:53:26.029 +08:00] [INFO] [server.go:326] ["start to server request"] [addr=http://172.16.12.169:8249]
[2023/04/06 16:53:26.030 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.16.221:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.17.65:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.18.27:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.16.221:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.18.27:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.17.65:8250]
[2023/04/06 16:53:29.544 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604087608672371]
[2023/04/06 16:53:32.547 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604088395104378]
[2023/04/06 16:53:35.650 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604089181536471]
[2023/04/06 17:03:26.021 +08:00] [INFO] [client.go:716] ["[sarama] Client background metadata update:kafka: no specific topics to update metadata"]
[2023/04/06 16:53:38.653 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604089967968464]

Old drainer startup log

[2023/04/06 16:53:13.786 +08:00] [INFO] [version.go:50] ["Welcome to Drainer"] ["Release Version"=v4.0.3-dirty] ["Git Commit Hash"=fb0cc0d5290202bb6d1b6f189549f629cfd810ef] ["Build TS"="2021-10-21 02:40:08"] ["Go Version"=go1.15.6] ["Go OS/Arch"=linux/amd64]
[2023/04/06 16:53:13.786 +08:00] [INFO] [main.go:46] ["start drainer..."] [config="{\"log-level\":\"info\",\"node-id\":\"172.16.18.27:
| username: qufudcj | Original post link

I don’t know if this
[“[sarama] Client background metadata update:kafka: no specific topics to update metadata”]
is related.

| username: qufudcj | Original post link

And using the Java consumer from GitHub to try to consume, it cannot read the messages. Using Kafka’s command line consumer also gets stuck and does not display any return.

| username: qufudcj | Original post link

Does this mean that the topic cannot be found? Do I need to delete and recreate it?

| username: qufudcj | Original post link

I changed to a new topic, but still no messages are coming in…

| username: qufudcj | Original post link

The configuration of drainer is:

- host: 172.16.12.169
  ssh_port: 40022
  port: 8249
  deploy_dir: /alidata/tidb/deploy
  data_dir: data
  config:
    syncer.db-type: kafka
    syncer.to.kafka-addrs: 172.16.16.221:9092,172.16.17.65:9092,172.16.18.27:9092
    syncer.to.kafka-max-message-size: 1610612736
    syncer.to.kafka-max-messages: 1536
    syncer.to.kafka-version: 1.0.2
    syncer.to.topic-name: dcj-test
  arch: amd64
  os: linux
| username: tidb狂热爱好者 | Original post link

Your TiDB is quite old, isn’t it?

| username: qufudcj | Original post link

Yes, 4.0.4. We haven’t upgraded because the production environment is also involved with binlog synchronization, so there has been no upgrade plan.

| username: db_user | Original post link

Check this out and review the status:

| username: qufudcj | Original post link

Currently, I see both are online. The top line is the newly added drainer, and the bottom one is the original.

| username: qufudcj | Original post link

I also tried marking the old one as offline, but there are still no new messages in the message queue.

| username: db_user | Original post link

Is the status of the pumper normal as well?

| username: qufudcj | Original post link

All are online

Currently, the display looks normal

I checked the old drainer logs, and there was no such error as “kafka: no specific topics to update metadata”.

| username: db_user | Original post link

This should be a bug, but it seems there is no solution:
remove confused sarama log kafka: no specific topics to update metadata · Issue #969 · pingcap/tidb-binlog (github.com)

| username: qufudcj | Original post link

I just tried to add another drainer and found that messages were being sent, but after a while, it reported an error and shut down by itself. Is it because the message is too large and Kafka didn’t respond within 30 seconds, causing a timeout?

[2023/04/07 09:45:16.393 +08:00] [INFO] [broker.go:212] ["[sarama] Connected to broker at 172.16.18.27:9092 (registered as #3)\n"]
[2023/04/07 09:45:19.622 +08:00] [INFO] [pump.go:166] ["receive big size binlog"] [size="592 MB"]
[2023/04/07 09:46:57.271 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440465904976789506]
[2023/04/07 09:46:57.271 +08:00] [ERROR] [syncer.go:457] ["Failed to close syncer"] [error="fail to push msg to kafka after 30s, check if kafka is up and working"] [errorVerbose="fail to push msg to kafka after 30s, check if kafka is up and working\ngithub.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:236\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:971] ["[sarama] producer/broker/3 state change to [closing] because write tcp 172.16.12.165:47234->172.16.18.27:9092: i/o timeout\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [broker.go:253] ["[sarama] Closed connection to broker 172.16.18.27:9092\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:578] ["[sarama] producer/leader/tidb-binlog-pre/0 state change to [retrying-1]\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:588] ["[sarama] producer/leader/tidb-binlog-pre/0 abandoning broker 3\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:717] ["[sarama] producer/broker/3 input chan closed\n"]
[2023/04/07 09:46:58.419 +08:00] [INFO] [async_producer.go:801] ["[sarama] producer/broker/3 shut down\n"]
[2023/04/07 09:46:58.919 +08:00] [INFO] [client.go:772] ["[sarama] client/metadata fetching metadata for [tidb-binlog-pre] from broker 172.16.17.65:9092\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:711] ["[sarama] producer/broker/3 starting up\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:727] ["[sarama] producer/broker/3 state change to [open] on tidb-binlog-pre/0\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:570] ["[sarama] producer/leader/tidb-binlog-pre/0 selected broker 3\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:594] ["[sarama] producer/leader/tidb-binlog-pre/0 state change to [flushing-1]\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:616] ["[sarama] producer/leader/tidb-binlog-pre/0 state change to [normal]\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [broker.go:212] ["[sarama] Connected to broker at 172.16.18.27:9092 (registered as #3)\n"]
[2023/04/07 09:47:07.272 +08:00] [INFO] [server.go:451] ["begin to close drainer server"]
[2023/04/07 09:47:07.272 +08:00] [ERROR] [util.go:65] ["Recovered from panic"] [err="\"Waiting too long for `Syncer.run` to quit.\""] ["real stack"="github.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1.1\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:67\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).run\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:463\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).Start\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:132\ngithub.com/pingcap/tidb-binlog/drainer.(*Server).Start.func4\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/server.go:288\ngithub.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:75"] [name=syncer]
[2023/04/07 09:47:07.272 +08:00] [INFO] [util.go:72] [Exit] [name=syncer]
[2023/04/07 09:47:07.275 +08:00] [INFO] [server.go:416] ["has already update status"] [id=172.16.12.165:8249]
| username: db_user | Original post link

Check the status and error logs of Kafka, it might be related to the configuration of message-bytes.

| username: qufudcj | Original post link

This value has already been set quite high, and Kafka does not have the “Message was too large” error.

Can drainer actively limit the message size?

| username: db_user | Original post link

You can adjust the configuration according to these two articles.

| username: qufudcj | Original post link

It seems that this option is not available in version 4.0.

| username: qufudcj | Original post link

I tried various operations, such as upgrading the Kafka version and creating a new queue, but it still didn’t work well.

I want to try replacing the binary file with a higher version of Drainer alone. What catastrophic consequences might this have?