I am seeing around 1 million logs per hour of:
[2023/10/16 18:46:39.464 +00:00] [WARN] [admin.go:100] ["query kafka cluster meta failed, retry it"] [namespace=default] [changefeed=tidb-replication-task] [error="kafka: tried to use a client that was closed"]
on TiCDC. This does not appear to affect operation at all. Is there a way that I can fix this or suppress this warn message? Ideally we do not want to ingest all these logs but still want to keep a record of other error messages in TiCDC.
The warning message you are seeing in TiCDC logs indicates that there is an issue with querying the Kafka cluster metadata. This warning does not affect the operation of TiCDC, but it can generate a large number of logs.
To fix this issue, you need to ensure that TiCDC can successfully query the Kafka cluster metadata. Here are a few steps you can take:
Check the connectivity between TiCDC and the Kafka cluster. Make sure that TiCDC can reach the Kafka brokers and that there are no network issues.
Verify the Kafka cluster configuration in the TiCDC configuration file. Ensure that the Kafka cluster address and other relevant settings are correctly specified.
Check the Kafka cluster status and health. Ensure that the Kafka brokers are running properly and that there are no issues with the cluster itself.
If you have confirmed that the Kafka cluster is functioning correctly and TiCDC can connect to it, but you still see the warning message, you can consider suppressing the warning message in the TiCDC logs. However, please note that this will also suppress other warning messages related to querying the Kafka cluster metadata.
To suppress the warning message, you can modify the log level configuration for TiCDC. By setting the log level to a higher level, such as “error” or “fatal”, you can filter out the warning messages while still keeping a record of other error messages in TiCDC.
You can modify the log level configuration in the TiCDC configuration file. Look for the log-level parameter and set it to the desired level. For example:
[log]
level = "error"
After making the changes, restart the TiCDC service for the configuration to take effect.
Please note that modifying the log level configuration should be done with caution, as it can affect the visibility of other important error messages. It is recommended to monitor the TiCDC logs closely to ensure that no critical error messages are being suppressed.
For more information on TiCDC configuration and log management, you can refer to the official TiCDC documentation .