TiDB GC Cleanup Exception Leading to Continuous Space Increase

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb gc清理异常,导致空间不断上涨

| username: tug_twf

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version] 5.1.4
[Reproduction Path] GC time cleanup is very slow
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]
GC cleanup has been stuck for several days
image

Checking the GC cleanup logs, there are continuous failures, and it seems the failures are all in the delete range

From the current cluster view, there doesn’t seem to be much physical deletion that needs to be cleaned up
image

GC related monitoring

| username: h5n1 | Original post link

Refer to this.

| username: DBRE | Original post link

In this scenario, can restarting TiKV free up space?

| username: h5n1 | Original post link

Give it a try, find a time to perform a rolling restart.

| username: DBRE | Original post link

Alright, alright.

| username: 像风一样的男子 | Original post link

You can upgrade the minor version to 5.4.

| username: TiDBer_小阿飞 | Original post link

Is this considered a version bug?

| username: tug_twf | Original post link

Restarting probably won’t solve the problem; we’ve already tried restarting.

| username: tug_twf | Original post link

However, our version is 5.1.4, and we are not sure if it has the same issue.

| username: hzc989 | Original post link

(I am the colleague of the original poster of this issue) To supplement the current suspected points and situation:

In the mysql.gc_delete_range table, there have been a few ranges that have been lingering since 0908, which is significantly different from other new ranges.

From the delete logs of the past month, it is also found that the deletion of these ranges has consistently failed:

| username: zhanggame1 | Original post link

Try modifying the GC time multiple times to see if it has any effect.

| username: ajin0514 | Original post link

Modify the GC event.

| username: 像风一样的男子 | Original post link

Version 5.3.1 fixed a bug that is very similar to the phenomenon you described.

| username: 路在何chu | Original post link

It is most likely a bug. GC cleanup is supposed to be very fast. I tested modifying tikv_gc_life_time, and when I reduced it, the previous data was deleted instantly.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.