Uneven Data Distribution in TiFlash

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash 数据量分布不均匀

| username: 友利奈绪

【TiDB Usage Environment】Production Environment
【TiDB Version】7.5
【Encountered Problem: Problem Phenomenon and Impact】
The data distribution in TiFlash is uneven. Two of the three nodes have tens of gigabytes of data, while the other has 200 gigabytes. How can I balance the data among the TiFlash nodes? Can any experts help me figure out how to solve this?

2
| username: 友利奈绪 | Original post link

Is the storage surge a GC issue? All other nodes are normal.

3
| username: lemonade010 | Original post link

Is there a mixed deployment? First, log in to the host to confirm if all the data is from TiFlash. Some of it might be log data causing the imbalance.

4
| username: 友利奈绪 | Original post link

A certain table suddenly ballooned in file size to 174GB, but the table size only shows 2GB.

5
| username: TiDBer_QKDdYGfz | Original post link

Are you saying that the GC didn’t delete the old data?

6
| username: lemonade010 | Original post link

Check the TiFlash logs for any related error information?

7
| username: 友利奈绪 | Original post link

Indeed, it is historical and has not been deleted. The normal TiFlash node is just one file.

8
| username: zhaokede | Original post link

Is GC enabled?

9
| username: 友利奈绪 | Original post link

It’s turned on.

10
| username: 友利奈绪 | Original post link

The reboot method solved it.

11
| username: 友利奈绪 | Original post link

Does anyone know the reason why it was resolved? Restarting fixed it.

12
| username: 我是吉米哥 | Original post link

The three basic IT troubleshooting steps: restart, reinstall, replace the machine. :sweat_smile:

13
| username: Kongdom | Original post link

:joy: Another possibility is that the data in the monitoring system is outdated. After restarting and fetching new data, it should be fine.