Backup to NFS Error: File or Directory Not Found

translator_bot · June 23, 2024, 8:00am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br 备份至nfs报错 File or directory not found

| username: OnTheRoad

The NFS shared folder has been mounted to /tidbbak on each TiKV node, and an empty folder /tidbbak/fullbak has been created.
The backup command is as follows:

br backup full --pd "192.168.3.221:2379" --storage "local://tidbbak/fullbak" --ratelimit 120 --log-file /tidbbak/fullbak/fullbackup_`date +%Y%m%d`.log

The error message is as follows (extra complaint: the log is too long, 140,000 words: my eyes are almost blind, only the last error message is intercepted):

["Full Backup failed summary"] [total-ranges=86] [ranges-succeed=0] [ranges-failed=86] [backup-total-ranges=86] [backup-total-regions=102] [unit-name="range ......

140,000 words omitted here

(*Group).Go.func1\
\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1571"]
Error: error happen in store 1 at 192.168.3.225:20160: local:<path:"/fullbak" >  File or directory not found on TiKV Node (store id: 1; Address: 192.168.3.225:20160). work around:please ensure br and tikv nodes share a same storage and the user of br and tikv has same uid.: [BR:KV:ErrKVStorage]tikv storage occur I/O error

No error occurs when backing up using the following script:

#!/bin/bash

export DATEDIR=`date +%Y%m%d`

export BASEDIR=/tidbbak

mkdir $BASEDIR/$DATEDIR

br backup full --pd "192.168.3.221:2379" --storage "local://$BASEDIR/$DATEDIR" --ratelimit 120 --log-file $BASEDIR/$DATEDIR/fullbackup_`date +%Y%m%d`.log

translator_bot · June 23, 2024, 8:00am

| username: xfworld | Original post link

Posted a bunch of logs but didn’t provide the most important information. I want to K you~~

Please specify the TiDB version and the BR version.

The logs describe a path that cannot be found, indicating a UID mismatch between the TiKV user and the BR user.

Error: error happen in store 1 at 192.168.3.225:20160: local:<path:“/fullbak” > File or directory not found on TiKV Node (store id: 1; Address: 192.168.3.225:20160). Workaround: please ensure BR and TiKV nodes share the same storage and the user of BR and TiKV has the same UID.

translator_bot · June 23, 2024, 8:00am

| username: banana_jian | Original post link

For NFS, it needs to be mounted on all TiKV and TiFlash nodes, and must have read and write permissions.

translator_bot · June 23, 2024, 8:00am

| username: ShawnYan | Original post link

Adjust the format of the post, is this a test display on the forum?

Additionally, the log level can be adjusted with --log-level.

translator_bot · June 23, 2024, 8:00am

| username: OnTheRoad | Original post link

This is the complete log. The log is one of my complaints.
The user’s UID is consistent.

translator_bot · June 23, 2024, 8:00am

| username: TiDBer_jYQINSnf | Original post link

Let me explain why the logs look so ugly. This is because it’s a panic in Go, which outputs the entire call stack. It means that this type of error wasn’t handled in the code. So the error looks bad. As for the specific reason, looking at these logs, it shows that getalltikvstores failed, which means the RPC call to PD failed. To understand why it was triggered, more logs would need to be posted.