Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: TIKV故障导致tidb-server无法启动
|
username: 巨化斑鸠
【TiDB Usage Environment】Production Environment
【TiDB Version】V2.1.10
【Reproduction Path】Due to TiKV disk being full, causing TiDB
【Encountered Problem: Symptoms and Impact】tidb-server cannot start
【Resource Configuration】Log Information
Startup error message:
ERROR MESSAGE SUMMARY ****************************************************************************************************************************************************************************************
[TiKV2-2]: Ansible FAILED! => playbook: start.yml; TASK: wait until the TiKV port is up; message: {“changed”: false, “elapsed”: 300, “msg”: “the TiKV port 20172 is not up”}
[TiKV1-2]: Ansible FAILED! => playbook: start.yml; TASK: wait until the TiKV port is up; message: {“changed”: false, “elapsed”: 300, “msg”: “the TiKV port 20172 is not up”}
[10.3.0.147]: Ansible FAILED! => playbook: start.yml; TASK: wait until the TiDB port is up; message: {“changed”: false, “elapsed”: 300, “msg”: “the TiDB port 4000 is not up”}
[10.3.0.149]: Ansible FAILED! => playbook: start.yml; TASK: wait until the TiDB port is up; message: {“changed”: false, “elapsed”: 300, “msg”: “the TiDB port 4000 is not up”}
[10.3.0.148]: Ansible FAILED! => playbook: start.yml; TASK: wait until the TiDB port is up; message: {“changed”: false, “elapsed”: 300, “msg”: “the TiDB port 4000 is not up”}
Information of the TiKV that cannot start
---------tikv_stderr.out----------
thread ‘main’ panicked at ‘invalid auto generated configuration file “/lsb/data2/deploy/data/last_tikv.toml”, err expected an equals, found eof at line 290’, src/config.rs:1122:17
stack backtrace:
0: std::sys::unix::backtrace::tracing:
:unwind_backtrace
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: std::panicking::default_hook::{{closure}}
at libstd/sys_common/backtrace.rs:71
2: std::panicking::default_hook
at libstd/panicking.rs:227
3: std::panicking::rust_panic_with_hook
at libstd/panicking.rs:475
4: std::panicking::continue_panic_fmt
at libstd/panicking.rs:390
5: std::panicking::begin_panic_fmt
at libstd/panicking.rs:345
6: tikv_server::main
at src/config.rs:1122
7: std::rt::lang_start::{{closure}}
at /checkout/src/libstd/rt.rs:74
8: main
9: __libc_start_main
10:
TiDB log information
2023/10/19 12:08:13.036 +08:00] [FATAL] [session.go:1368] [“check bootstrapped failed”] [error=“[tikv:9005]Region is unavailable[try again later]”] [errorVerbose="[tikv:9005]Region is unavailable[try again later]
Check ps process
[tidb@TiDB-test09 ~]$ ps -ef|grep tidb-server
tidb 29604 1 1 12:17 ? 00:00:00 bin/tidb-server -P 4000 --status=10080 --advertise-address=10.3.0.147 --path=10.3.0.147:2379,10.3.0.148:2379,10.3.0.149:2379 --config=conf/tidb.toml --log-slow-query=/lsb/tidb/debloy/log/tidb_slow_query.log --log-file=/lsb/tidb/debloy/log/tidb.log
tidb 29639 29560 0 12:18 pts/2 00:00:00 grep --color=auto tidb-server
[tidb@TiDB-test09 ~]$ netstat -an|grep 4000
[tidb@TiDB-test09 ~]$
|
username: 
|
username: 
|
username: