Using TiKV as JuiceFS Metadata: JuiceFS Data Unusable and Size Discrepancy After Backup with TiKV-BR Backup Raw

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用tikv当juicefs的元数据，使用使用tikv-br backup raw 备份数据后，备份的数据juicefs无法使用，数据量大小也明显不一样

| username: TiDBer_Fi87GTmL

[Test Environment for TiDB] Testing
[TiDB Version] 6.5.4
[Reproduction Path] Backup and Restore
[Encountered Issue: Problem Phenomenon and Impact]

  1. Using TiKV as metadata storage for JuiceFS
  2. Backup TiKV
sh backup.sh
Checking updates for component tikv-br... Timedout (after 2s)
Starting component tikv-br: /home/tidb/.tiup/components/tikv-br/v1.1.0/tikv-br backup raw --pd 10.21.9.28:2379 --storage s3://tidb-backup/juicefs-recovery-verify/juicefs-pro-backup-20240304-114051 --s3.endpoint http://10.21.9.31 --log-file log/backupraw-juicefs-pro-20240304114051.log --gcttl=2h --format=raw
Detail BR log in log/backupraw-juicefs-pro-20240304114051.log
Raw backup <------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2024/03/04 11:40:59.103 +08:00] [INFO] [collector.go:67] ["Raw backup success summary"] [total-ranges=344] [ranges-succeed=344] [ranges-failed=0] [backup-total-regions=372] [total-take=5.231582018s] [total-kv=2623767] [total-kv-size=7.289GB] [average-speed=1.393GB/s] [backup-data-size(after-compressed)=821.4MB]
  1. Restore data
Starting component tikv-br: /home/tidb/.tiup/components/tikv-br/v1.1.0/tikv-br restore raw --pd 10.21.8.33:2379 --s3.endpoint http://10.21.9.31 --storage s3://tidb-backup/juicefs-recovery-verify/juicefs-pro-backup-20240304-114051 --log-file restoreraw.log
Detail BR log in restoreraw.log
Raw Restore <-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2024/03/04 11:41:52.009 +08:00] [INFO] [collector.go:67] ["Raw restore success summary"] [total-ranges=344] [ranges-succeed=344] [ranges-failed=0] [restore-files=344] [total-take=5.044049245s] [restore-data-size(after-compressed)=821.4MB] [total-kv=2623767] [total-kv-size=7.289GB] [average-speed=1.445GB/s]
  1. JuiceFS connects to the restored data
juicefs status tikv://10.21.8.33:2379,10.21.8.34:2379,10.21.8.35:2379/juicefs_prd50
2024/03/04 11:49:22.755933 juicefs[720341] <INFO>: Meta address: tikv://10.21.8.33:2379,10.21.8.34:2379,10.21.8.35:2379/juicefs_prd50 [interface.go:497]
2024/03/04 11:49:22.756290 juicefs[720341] <INFO>: TiKV gc interval is set to 3h0m0s [tkv_tikv.go:84]
2024/03/04 11:49:22.772405 juicefs[720341] <FATAL>: load setting: database is not formatted, please run `juicefs format ...` first [status.go:96]
2
| username: WalterWj | Original post link

JuiceFS uses raw KV, and it seems that the TiDB official website does not provide a raw KV backup tool :thinking:.

3
| username: TiDBer_Fi87GTmL | Original post link

Here it is: TiKV | RawKV BR 使用手册

4
| username: WalterWj | Original post link

Excellent.

5
| username: 有猫万事足 | Original post link

It didn’t get the data here.
Scroll down,

The key’s name is setting.

The implementation of getting a key is here. I think you can use the client-go package to write a piece of code yourself and see if you can get the setting key on both clusters.

From the tikv-br backup logs, there really doesn’t seem to be any issue. The size and the number of keys match.
Overall, it still feels like a configuration issue. It’s probably not a backup and restore problem.

6
| username: dba远航 | Original post link 
export AWS_ACCESS_KEY_ID=${AWS_KEY_ID};
export AWS_SECRET_ACCESS_KEY=${AWS_KEY};
tikv-br backup raw \
    --pd="${PDIP}:2379" \
    --storage="s3://backup-data/2022-09-16/" \
    --dst-api-version v2 \
    --log-file="/tmp/backupraw.log" \
    --gcttl=5m \
    --start="a" \
    --end="z" \
    --format="raw"
7
| username: redgame | Original post link

tikv-br backup raw --help

8
| username: 小于同学 | Original post link

tikv-br backup raw --help