TiKV Storage Directory Imbalance

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiKV存储目录不均衡

| username: TiDB_C罗

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

TiKV-Detail


TiDB Dashboard


View system directory


What is the reason for this?

| username: dba-kit | Original post link

It is indeed strange. The number of regions is the same across the three nodes, but one TiKV node has a larger data volume. Can you check if the GC on the TiKV with the larger data volume is functioning normally?

| username: dba-kit | Original post link

Go into the db directory and check which directory has the largest amount of data?

| username: tidb菜鸟一只 | Original post link

Check if the logs in the database are taking up a lot of space.

| username: h5n1 | Original post link

Check the size differences in the db directory. Are the two smaller ones missing the place_holder file?

| username: 表渣渣渣 | Original post link

Is it a problem with the logs?

| username: 表渣渣渣 | Original post link

Go to the tikv directory and use du -sh ./* to compare the main difference files of the three.

| username: TiDB_C罗 | Original post link

Not much

| username: TIDB-Learner | Original post link

If it is not deployed together with other modules, there may be hotspot issues and uneven region distribution.

| username: 这里介绍不了我 | Original post link

It can be ruled out that uneven Region distribution is the cause. Consider whether it is due to untimely GC.

| username: 小龙虾爱大龙虾 | Original post link

The difference is not that much. Check the region size panel.

| username: yytest | Original post link

Indeed, it is necessary to look at the underlying logs to determine the issue.

| username: TiDBer_QYr0vohO | Original post link

Not much difference, it’s okay.

| username: yytest | Original post link

Do you have any relevant logs?

| username: 小于同学 | Original post link

Not much difference.

| username: TiDBer_rvITcue9 | Original post link

Most likely it’s the log.

| username: 友利奈绪 | Original post link

If log files are not set with cleanup rules, they can become very large.

| username: zhanggame1 | Original post link

Check if the log sizes are different.

| username: TiDB_C罗 | Original post link

How to set up cleaning, I don’t dare to touch the files under the db directory.

| username: zhaokede | Original post link

Look at the specific directory to see the differences between data files and log files.