TiFlash cannot start when installed under mounted NAS

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiflash在挂载NAS下安装不能启动

| username: 文成Yu

[TiDB Usage Environment] Test Environment
[TiDB Version] v6.1.2

Alibaba Cloud ECS mounted NAS, installed TiDB, TiFlash couldn’t start, logs are as follows:

[2022/11/06 20:24:45.236 +08:00] [WARN] [server.rs:1418] ["check: rocksdb-data-dir"] [err="config fs: data-dir.rotation.get: \"0084b86b-4rkk.cn-hangzhou.extreme.nas.aliyuncs.com:/share\" no device find in block"] [path=/mnt/tidb-data/tiflash-9000/flash]
[2022/11/06 20:24:45.236 +08:00] [INFO] [config.rs:879] ["data dir"] [mount_fs="FsInfo { tp: \"nfs\", opts: \"rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,noacl,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.94.205,mountvers=3,mountport=2049,mountproto=tcp,local_lock=all,addr=172.16.94.205\", mnt_dir: \"/mnt/tidb-data\", fsname: \"0084b86b-4rkk.cn-hangzhou.extreme.nas.aliyuncs.com:/share\" }"] [data_path=/mnt/tidb-data/tiflash-9000/flash/raft]
[2022/11/06 20:24:45.236 +08:00] [WARN] [server.rs:1426] ["check: raftdb-path"] [err="config fs: data-dir.rotation.get: \"0084b86b-4rkk.cn-hangzhou.extreme.nas.aliyuncs.com:/share\" no device find in block"] [path=/mnt/tidb-data/tiflash-9000/flash/raft]

[2022/11/06 20:24:45.240 +08:00] [WARN] [server.rs:523] ["failed to remove space holder on starting: No such file or directory (os error 2)"]
[2022/11/06 20:24:47.078 +08:00] [FATAL] [lib.rs:468] ["Failed to reserve space for recovery: Operation not supported (os error 95)."] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n   1: std::panicking::rust_panic_with_hook\n   2: std::panicking::begin_panic_handler::{{closure}}\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n   4: rust_begin_unwind\n   5: core::panicking::panic_fmt\n   6: server::server::run_tikv\n   7: server::proxy::run_proxy\n   8: _ZN2DB20RaftStoreProxyRunner20runRaftStoreProxyFFIEPv\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/dbms/src/Server/Server.cpp:462:9\n   9: start_thread\n  10: __GI___clone\n"] [location=components/server/src/server.rs:533] [thread_name=<unnamed>]
2
| username: xfworld | Original post link

err=“config fs: data-dir.rotation.get: "0084b86b-4rkk.cn-hangzhou.extreme.nas.aliyuncs.com:/share" no device find in block”

This means the device cannot be found, so it definitely won’t start…

Are you sure the mounted directory is accessible?

3
| username: 文成Yu | Original post link

The NAS is mounted correctly.
Using tiup cluster deploy, the installation can proceed normally, and data files are generated after installation.
The error occurs only when running tiup cluster start.

4
| username: xfworld | Original post link

It doesn’t support this mounting mode, right?

config fs: data-dir.rotation.get: “0084b86b-4rkk.cn-hangzhou.extreme.nas.aliyuncs.com:/share”
no device found in block

Isn’t there TiDB Cloud on Alibaba Cloud? You can use it directly.

5
| username: 文成Yu | Original post link

Resolved by modifying the configuration:

server_configs:
  tiflash-learner:
    storage.reserve-space: 0MB
6
| username: alfred | Original post link

How is the performance when using NAS? After all, it involves an additional network I/O.

7
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.