Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: tiflash 问题 inode 满了
|
username: kuuhaku
【TiDB Usage Environment】Production Environment
【TiDB Version】5.3.3
【Reproduction Path】No operations
【Encountered Problem: Problem Phenomenon and Impact】
Preliminary suspicion that the tiflash data directory has a large number of subdirectories causing the inode to be full
【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】
Expand it. There are too many small files in TiFlash, especially when there are many partitions.
There’s nothing much to avoid, it’s very sudden. We haven’t really used TiFlash much on our side. This cluster has only been in use for about a year and it’s already like this.
I haven’t used TiFlash, but I noticed that the data directory creates a new batch of directories and files over time. Is this normal?
If you are not using TiFlash, you can scale it down. The generated files should be related to schema synchronization.
In the past few months, I have been using it, but to be honest, scaling out is a bit challenging because there is still plenty of hard disk space. Are there other ways to address this issue?
I see that a new file is generated each time, and the old files are still retained. Is this normal?
Find out which table occupies the most inodes,
SELECT * FROM INFORMATION_SCHEMA.TIFLASH_TABLES a WHERE a.table=‘t_1615’;
No, t_xxxx has many such directories, but TIFLASH_TABLES is empty.
SELECT * FROM INFORMATION_SCHEMA.TIFLASH_SEGMENTS a—Is this table also empty?
Starting from version 6.1, you can manually compact and clean up historical useless data in TiFlash, but it seems that it was automatically cleaned up before.
The version might be a bit low. I feel that there might be historical useless data that hasn’t been properly cleaned up. Are there a lot of files in the t_xxxx directories under /tidb/tidb-data/tiflash-9000/data/, or are there just a lot of t_xxxx directories themselves?
The directory of t_xxxx has a lot of files. Running ll takes a long time, and wc -l directly freezes.
Is there a lot of data in this table? This table is for those that actually use TiFlash replicas.
You haven’t used TiFlash. To use TiFlash for your table, you need to add a TiFlash replica:
ALTER TABLE sbtest1 SET tiflash replica 1;
If the table above doesn’t have any data, I think you can scale down TiFlash.
It’s likely that in your version of TiFlash, even if you haven’t added a TiFlash replica, it still creates TiFlash metadata for the table by default. Therefore, there will be a lot of metadata information in the TiFlash directory, but very little data information. This results in a situation where many inodes are used, but very little space is consumed.
I guess you have many TiKV nodes and many tables in the database, but very few TiFlash nodes.
Yes, there are many database tables, 6 KV nodes, but only two TiFlash nodes.
To prevent this issue, can we only upgrade the TiDB version? Do you have any recommended versions?
If you use TiFlash, I think you can use a relatively new version because there have been quite a few optimizations in the recent versions of TiFlash. I have two versions on hand, one is 5.4.3 which is similar to yours, and the other is 6.5. Only when the TiFlash replica is set will the metadata be saved.
|
username: 
|
username: