Network Failure Causes Log Accumulation, 3TB Disk Fills Up Immediately

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 网络故障导致日志堆积,3T磁盘直接打满

| username: grafanab

[TiDB Usage Environment] Production Environment
[TiDB Version]
[Reproduction Path] tidb/tikv-20160/log
Operations performed that caused the issue:
Issue Description: Multiple TiDB nodes lost network connectivity, causing the cluster to be unable to communicate. This resulted in continuous logging in the tidb/tikv-20160/log directory, generating over 10,000 log files, with each file averaging 301M. The 3T+ disk space was completely filled.
[Encountered Issue: Symptoms and Impact]
[Resource Configuration] Navigate to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]


| username: tidb菜鸟一只 | Original post link

Has the network been restored? If it has, will deleting the logs immediately free up space?

| username: h5n1 | Original post link

TiKV log.file configuration item

| username: 像风一样的男子 | Original post link

This kind of log can be directly removed with rm.

| username: xingzhenxiang | Original post link

Logs with timestamps can be directly removed.

| username: zhaokede | Original post link

It is indeed unscientific. If the same type of log issue is not resolved, the frequency should be reduced later.

| username: grafanab | Original post link

I don’t dare to delete it. I tested it before, and deleting everything will cause the node to fail to restart properly.

| username: grafanab | Original post link

Indeed, it’s not scientific; there are no logs of the same type.

I tried before, deleting all logs caused the node to fail to restart. There should be some logs that cannot be deleted.

| username: h5n1 | Original post link

Are you sure it’s the .log file that was deleted? Just keep the current tikv.log.

| username: 江湖故人 | Original post link

The log currently being written is generally filled with echo > x.log, other logs can be deleted directly.

| username: Inkjade | Original post link

Keep the current log being written, tikv.log. You can delete the other logs.

| username: zhanggame1 | Original post link

Impossible, did you delete the wrong thing?

| username: 小龙虾爱大龙虾 | Original post link

The consequences of not adding log file-related parameters: when deleting, the current logs need to be retained.

| username: 像风一样的男子 | Original post link

Delete the ones with dates
rm -rf *2024*.log like this

| username: oceanzhang | Original post link

Is this TiKV’s own runtime log?

| username: wangccsy | Original post link

Manually remove it with RM.

| username: dba远航 | Original post link

Manually delete it. If it continues, write a script to delete it periodically.

| username: 哈喽沃德 | Original post link

The official team should optimize this issue. When duplicate logs reach a certain threshold, the logging interval should automatically extend to a specified time.

| username: changpeng75 | Original post link

Is it not possible to perform log dumping and rotation reuse?

| username: 这里介绍不了我 | Original post link

Do not delete the current writes, just clean up the historical ones.