How to Expand TiKV Disk Capacity When Insufficient

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIKV磁盘不足,如何扩容

| username: TiDBer_rZHKXZxU

【TiDB Usage Environment】Production Environment
【TiDB Version】7.5
【Reproduction Path】What operations were performed when the issue occurred
【Encountered Issue: Issue Phenomenon and Impact】
【Resource Configuration】Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】

There are four TiKV machines, and the disk usage has reached 85%, starting to alarm.
Three of them use data disks that can be infinitely expanded, but one is directly deployed in the root directory of the system disk, which cannot be expanded further, with a maximum of 2TB. How should this situation be handled?

| username: DBAER | Original post link

  1. You can add another node for data balance, and then delete the node.
  2. Add another node, mount a data disk, stop the TiKV on the node with insufficient disk space, create a symbolic link for the directory, and then restart TiKV.
    Additionally, testing and verification are required.
| username: GreenGuan | Original post link

The scenario I understand is that tikv1, 2, and 3 use mounted disks (similar to LVM), while tikv4 uses local disks (which cannot be expanded).

In this scenario, it is more effective to perform physical expansion of the LVM on tikv1, 2, and 3 (first making space for tikv4’s data), then shrink tikv4, mount the data disk to tikv4 after shrinking, and finally scale-out tikv4.

| username: TiDBer_RjzUpGDL | Original post link

Scale down TiKV4, expand the disk, change the path, and then scale up TiKV4 again.

| username: TiDBer_H5NdJb5Q | Original post link

May I ask, can I only use a soft link? Can’t I just copy the file and then change the data storage directory?

| username: TiDBer_rZHKXZxU | Original post link

Then I’ll add another node. After the data is synchronized, I’ll delete that node. The deletion of this node should be done in the management console, right? It’s been over 2 years since deployment and I haven’t touched it, so I’ve forgotten how to scale out.

| username: tidb菜鸟一只 | Original post link

Just use tiup scale-in to scale in.

| username: TIDB-Learner | Original post link

If the data is on the local disk, you can scale it down. If you still want to ensure 4 KV instances, you can scale up again to 4 KVs. You can also scale up first and then scale down the local disk KV.

| username: 随缘天空 | Original post link

First, add a new TiKV node. After it succeeds, you can scale down the machine where the system disk is deployed in the root directory.

| username: TiDBer_QYr0vohO | Original post link

You can add new nodes for expansion.

| username: paulli | Original post link

It is recommended to adjust the directory size of each node to be consistent.

| username: zhaokede | Original post link

Scale down the incorrectly deployed one, change the data directory, and scale up.

| username: YuchongXU | Original post link

Add a new node or expand the file system of the operating system.

| username: yytest | Original post link

You can try adding a TiKV node.

| username: 小于同学 | Original post link

Squatting for replies

| username: 友利奈绪 | Original post link

Just resize it again and it will be fine.

| username: TiDBer_rvITcue9 | Original post link

First, expand with a new 4-node cluster, then shrink and expand nodes 1, 2, and 3 one by one.

| username: zhh_912 | Original post link

Create an LVM logical volume, it’s easier to expand this way.

| username: Jack-li | Original post link

Migrate to other nodes.

| username: TiDBer_rZHKXZxU | Original post link

After expanding with a new node, data is currently being synchronized. The total data is 1.8T, and the synchronization seems to be very slow.