Log Accumulation Issue

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 日志堆积问题

| username: 健康的腰间盘

[TiDB Usage Environment] Production Environment
[TiDB Version] 7.5
[Reproduction Path] Default installed cluster
[Encountered Issue: Phenomenon and Impact] The tidb component log on one machine is too large
[Resource Configuration]


The disk usage on the first machine in the monitoring interface is too large, and it was found to be a tidb log issue.
Below are the logs from the problematic server
Problematic Server
Abnormal Log
Below are the logs from the normal server
Normal Server
Normal Log
Checked show config where name like ‘%log.%’
The log.file.max-days parameter on both machines is 0
Why is there log accumulation on only the first machine (approximately 70G of logs generated)?

| username: Defined2014 | Original post link

Check the logs to see what they are. There might have been a network issue or something similar during that time, causing a lot of logs to be generated. If they are not needed, you can manually delete the logs.

| username: 健康的腰间盘 | Original post link

It contains some SQL operation information, it should not be a network issue, the logs span from March of this year to the present.

| username: 这里介绍不了我 | Original post link

Is your TiDB connection direct or allocated through something like LVS?

| username: 健康的腰间盘 | Original post link

Direct connection

| username: Defined2014 | Original post link

That might be because the load balancing wasn’t done properly, and everything is running on one TiDB instance.

| username: 健康的腰间盘 | Original post link

Yes! Thank you.

| username: 这里介绍不了我 | Original post link

It is recommended to add a load balancer on top and allocate based on CPU capacity. Additionally, manually delete those redundant historical files.

| username: TiDBer_7S8XqKfl-1158 | Original post link

It feels like only one TiDB node is working, and the other nodes have no logs at all. This is definitely not right.

| username: TiDBer_TQXaqJ6U-6236 | Original post link

It feels like everything is on one host, there might be an issue with load balancing.

| username: TiDBer_7S8XqKfl-1158 | Original post link

Check if it’s a load balancing issue.

| username: TiDBer_3Cusx9uk-0775 | Original post link

What load balancer are you using? It seems like the load balancing has failed.

| username: 濱崎悟空 | Original post link

Create a load balancer.