How to Quickly Free Up Space on TiKV Nodes

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIKV节点如何快速释放空间

| username: TiDBer_pkQ5q1l0

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.0.0
The space on the TiKV node is insufficient. A large table was truncated in the morning, but the space was not released. Now it reports a “tikv is full” error.


| username: buddyyuan | Original post link

TiKV-Details → Please share the monitoring of the GC panel.

| username: TiDBer_pkQ5q1l0 | Original post link

Sorry, I can’t translate images. Please provide the text you need translated.

| username: wuxiangdong | Original post link

Merging empty regions

| username: TiDBer_pkQ5q1l0 | Original post link

Last time I increased the scheduling parameters, and as a result, TiKV crashed.

| username: tracy0984 | Original post link

Did TiKV experience an OOM (Out of Memory) issue?

| username: TiDBer_pkQ5q1l0 | Original post link

Currently, there is none.

| username: Jiawei | Original post link

TiDB does not immediately release space when data is deleted; it waits until the GC life time has passed to release the space. If possible, consider shortening the GC life time.

| username: TiDBer_pkQ5q1l0 | Original post link

The GC is already running every 10 minutes, but the number of regions is not decreasing at all.

| username: Jiawei | Original post link

Take a look to see if it’s confirmed that GC hasn’t reclaimed the space, or if other tables are occupying it.

| username: TiDBer_pkQ5q1l0 | Original post link

The table being truncated is the largest one, with over a billion rows of data. Logically, the space release should be noticeable. However, the capacity has not been released yet.

| username: buddyyuan | Original post link

It looks like the GC hasn’t progressed.

Take a look at the tidb->gc->Delete Range Failure OPM monitoring.

| username: TiDBer_pkQ5q1l0 | Original post link

I think the problem is that the tidb-server process is not running. You can check the status of the tidb-server process with the following command:

ps -ef | grep tidb-server

If the process is not running, you can start it with the following command:

/path/to/tidb-server --config=/path/to/tidb.toml
| username: buddyyuan | Original post link

It’s a bit strange, check the tikv logs.

| username: TiDBer_pkQ5q1l0 | Original post link

Are there any keywords that can be searched?

| username: buddyyuan | Original post link

According to this document, you can perform a simple preliminary check.

| username: 裤衩儿飞上天 | Original post link

Check the logs to see if the GC was successful.

| username: dba-kit | Original post link

Have you used TiCDC before?

| username: dba-kit | Original post link

The longest it can be blocked is 24 hours.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.