Error "No space left on device", TiFlash cannot start

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 报No space left on device ，tiflash 无法启动

| username: h5n1

[Version] V5.2.3 arm kylin-v10
[Issue]
tiflash cannot start, tiflash_err.log reports the following error, unable to create tiflash.log file, attempting to touch any file reports no space left Logging information to /tiflash/deploy/log/tiflash.log Poco::Exception. Code: 1000, e.code() = 28, e.displayText() = File access error: no space left on device: /tiflash/deploy/log/tiflash.log, e.what() = File access error Logging information to /tiflash/deploy/log/tiflash.log Poco::Exception. Code: 1000, e.code() = 28, e.displayText() = File access error: no space left on device: /tiflash/deploy/log/tiflash.log, e.what() = File access error

$ touch tiflash.log
touch: cannot touch ‘tiflash.log’: No space left on device
[tidb@pd151 log]$ touch 123
touch: cannot touch ‘123’: No space left on device

df shows 132G free
/dev/mapper/rootvg-lv_tiflash 147G 8.1G 132G 6% /tiflash

du -sh /tiflash

8.0G /tiflash

2
| username: h5n1 | Original post link

Check inode usage is full. Due to the database having partition tables created daily for each month and being dropped (126 tables per month), there are many historical DDLs and a large amount of metadata for historical tables in the data directory. In reality, TiFlash only synchronizes 8 tables.

# df -i /tiflash
Filesystem                     Inodes   IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-lv_tiflash 9830400 9830400     0  100% /tiflash
3
| username: h5n1 | Original post link

Checked and confirmed that the tables in the tiflash data directory currently exist and are not historical DDLs. The information inside pertains to the current tables in the database. Adding two months’ worth of partitions today increased the number of partitions and metadata, consuming a large number of inodes.

4
| username: h5n1 | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.