Cluster Abnormal Crash Using tiup Command Reports Error: mkdir /tmp/4767475860915002421: read-only file system

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 集群异常宕机使用tiup命令报错Error: mkdir /tmp/4767475860915002421: read-only file system

| username: Jolyne

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed to encounter the issue
Physical machine storage was damaged, after repair, starting the machine using tiup reports an error: Error: mkdir /tmp/4767475860915002421: read-only file system
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]

| username: 像风一样的男子 | Original post link

There is an issue with the disk. Check the status of the mounted disk. Use the mount command to see if the remarks indicate “ro”.

| username: Jolyne | Original post link

Are you looking at the data storage directory /dev/sdb1?

| username: tidb菜鸟一只 | Original post link

Can you manually create a directory under tmp to see if it can be created?

| username: Jolyne | Original post link

Tried it, can’t create, and the same error occurs.

| username: 像风一样的男子 | Original post link

Use df -h to check which disk partition is read-only, then unmount and remount it to see if it resolves the issue.

| username: 像风一样的男子 | Original post link

Additionally, is this control machine independent? If it’s urgent, I suggest migrating tiup to a new machine to ensure the cluster is fine, and then fix the control machine later.

| username: wzf0072 | Original post link

Check the folder permissions with ll /:

drwxrwxrwt.  33 root root 4096 Sep 1 16:57 tmp
| username: TiDBer_oHSwKxOH | Original post link

The disk is probably damaged.

| username: Jolyne | Original post link

There are no issues with permissions; I guess the disk on the IT side hasn’t been fixed.

| username: Jolyne | Original post link

Yes, just one central control machine. I’m planning to do that and move it to the new central control machine.

| username: Jolyne | Original post link

Is this also a disk failure?

| username: 像风一样的男子 | Original post link

Reporting “Read-only file system” usually indicates a disk problem.

| username: Jolyne | Original post link

Crashed, after mentioning last time that the SST file was corrupted and IT said there was a disk issue, it wasn’t fixed, and today everything went down :sweat:

| username: zhanggame1 | Original post link

Physical machine or virtual machine, what type of disk is it?

| username: redgame | Original post link

Disk issues or permissions

| username: dba-kit | Original post link

Your data disk is mounted in read-only mode. Check what is written in /etc/fstab? Or first unmount /dev/sdb1, then remount it and see what error it reports.

| username: zhanggame1 | Original post link

You might consider changing the machine; repairing it is too troublesome.

| username: Fly-bird | Original post link

No permission for this disk, please check the disk.