Truncate 100GB of Data but Disk Space Not Released

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: truncate100G数据 但磁盘空间没有释放

| username: Ann_ann

[TiDB Usage Environment] Production Environment
[TiDB Version] 3.0.6

  1. One of the TiKV nodes has insufficient disk space and cannot be expanded. I want to release disk space by truncating data. How long will it take approximately?
    This is the current GC mechanism time. It’s been half an hour, and the space has not been released.
  2. If I adjust the weight of this TiKV node, for example, set it to 0.5, will it release space?
| username: zhanggame1 | Original post link

Your gc time tidb_gc_life_time is 1 hour, and GC will be performed one hour after truncate.

To check the GC status, run select * from mysql.tidb. Anything before the tikv_gc_safe_point time has been GC’d.

| username: forever | Original post link

Wait for an hour to check, or reduce the tidb_gc_life_time a bit.

| username: 小龙虾爱大龙虾 | Original post link

At least 1 hour later to release, because your GC life time is 1 hour.

| username: Ann_ann | Original post link

Doesn’t tikv_gc_last_run_time indicate that a cleanup has been done?

| username: 有猫万事足 | Original post link

Cleared, but only up to 10:10, which is the time point below that line.

| username: 像风一样的男子 | Original post link

The reserved snapshot time is 10:10:01. If you execute it after 10 o’clock, you still have to wait.

| username: Ann_ann | Original post link

It’s been an hour, and the disk space still hasn’t been reclaimed.

| username: 像风一样的男子 | Original post link

Alright, I just realized you’re using 3.0. Starting from 4.0, it releases after GC. I’m not sure about the 3.0 version…

| username: tidb菜鸟一只 | Original post link

Even after truncating, you still need to compact to release the space.

| username: 裤衩儿飞上天 | Original post link

You need to wait for RocksDB’s GC to complete before it can be released.

| username: 随缘天空 | Original post link

Since all the data has been cleared, I think you can delete the table and then create an identical table to try. Then check if the disk space is released after 1 hour.

| username: zhanggame1 | Original post link

The data being cleaned up is from one hour after the truncate operation is completed, so you need to check tikv_gc_safe_point. Only the truncate operations before this point in time will be garbage collected.

| username: h5n1 | Original post link

Check how many entries are in mysql.delete_range and delete_range_done. Then check the TiDB logs of the GC leader for any busy-related information.

| username: Jellybean | Original post link

Question 1:
After truncating the table, you need to wait for the GC to execute to complete the space release. Based on the GC life time you posted, which is 1 hour, you should check again after at least 1 hour. You can also adjust this duration; the default value is usually 10 minutes. You can temporarily adjust it to speed up the GC and then revert it back afterward.

Question 2:
Releasing space after deleting data is automatically handled by GC and RocksDB Compaction in the background, so there’s no need to specifically adjust TiKV parameters. If the TiKV node’s disk is insufficient and cannot be expanded, you can only delete some logs and data. As for adjusting the “weight” you mentioned, I guess you are referring to PD scheduling regions, leaders, or scores. Adjusting these hastily is not beneficial for the overall cluster scheduling.

Moreover, if one TiKV node’s disk is insufficient, I suspect the disk space of other TiKV nodes is also nearly insufficient. Lowering the weight of this TiKV node will cause data to migrate to other nodes, which could potentially lead to bigger issues.

Additionally, the stable version of TiDB is now at v7.1, and v3.0 is really too old. I strongly recommend finding an opportunity to push for an upgrade, as there are substantial improvements in performance, stability, and feature richness.

| username: Ann_ann | Original post link

Thank you for the suggestion. It currently appears that after reclaiming three TiKV nodes, only one node is full. This node has a relatively high leader count.

| username: zxgaa | Original post link

The release happens after GC. TiDB’s truncate is different from MySQL’s.