TiDB Write Error: TiKV Disk Full, Monitoring Shows 100GB Available Space

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb写入报tikv 磁盘满了，实际监控看有100G的空间

| username: gejibin

When inserting data, the following error occurs:
insert into t values (1);
ERROR 1105 (HY000): tikv disk full: store_id:1 store_id:4 reason:“propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=Normal”

The actual Grafana monitoring shows 100G of data (with a size of 2T), but an error is reported when writing. Could you please advise what threshold will trigger this error? The database is not set to read-only at this time, right?

2
| username: WalterWj | Original post link

:thinking: Have you configured the capacity for TiKV?

3
| username: gejibin | Original post link

tikv 2T

4
| username: gejibin | Original post link

TiKV 配置文件描述 | PingCAP 文档中心, TiDB will reserve 5% of the space by default.

5
| username: 我是咖啡哥 | Original post link

Thank you @gejibin

When TiKV starts, it reserves a portion of space to protect disk space. When the remaining disk space is less than this reserved space, TiKV will limit some write operations.

The size of the reserved space is calculated as the maximum value between 5% of the disk capacity and the reserve-space configuration item (default: 5G).

Currently, the TiKV disk space is 2T, and 5% is exactly around 100G. This has triggered disk space protection.

6
| username: h5n1 | Original post link

Is it possible that the inodes are exhausted?

7
| username: tidb菜鸟一只 | Original post link

With only 100GB left out of 2TB, you still dare to write data? I suggest setting up an alert for 90% usage. When it reaches that point, you need to quickly find a solution…

8
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.