TiKV Crash or Panic

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv crash or panic

| username: TiDBer_KugsyWcF

【TiDB Usage Environment】Production Environment
【TiDB Version】7.1.3
【Reproduction Path】Unknown how to reproduce, no specific operations
【Encountered Problem: Phenomenon and Impact】tikv pod restart
【Resource Configuration】tikv 16C64G
【Attachments: Screenshots/Logs/Monitoring】
Logs:

[2024/03/07 17:59:29.043 +08:00] [ERROR] [time.rs:374] ["monotonic time jumped back, 1581214.068 -> 1581214.057"]
[2024/03/07 17:59:29.043 +08:00] [ERROR] [time.rs:374] ["monotonic time jumped back, 1581214.068 -> 1581214.057"]
[2024/03/07 17:59:29.045 +08:00] [FATAL] [lib.rs:510] ["elapsed=56976988; when=56976976"] [backtrace="  
 0: tikv_util::set_panic_hook::{{closure}}\n             at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:509:18
 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9
    std::panicking::rust_panic_with_hook
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13
 2: std::panicking::begin_panic_handler::{{closure}}
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:579:13
 3: std::sys_common::backtrace::__rust_end_short_backtrace
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18
 4: rust_begin_unwind
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:575:5
 5: core::panicking::panic_fmt
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:65:14
 6: tokio_timer::wheel::Wheel<T>::poll
 7: tokio_timer::timer::Timer<T,N>::process
    at rust/git/checkouts/tokio-8e927faba632ed16/4394380/tokio-timer/src/timer/mod.rs:272:33
    <tokio_timer::timer::Timer<T,N> as tokio_executor::park::Park>::park
    at rust/git/checkouts/tokio-8e927faba632ed16/4394380/tokio-timer/src/timer/mod.rs:379:9
    tokio_timer::timer::Timer<T,N>::turn
    at rust/git/checkouts/tokio-8e927faba632ed16/4394380/tokio-timer/src/timer/mod.rs:256:21
 8: tikv_util::timer::start_global_steady_timer::{{closure}}
    at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/timer.rs:209:17
    <std::thread::Builder as tikv_util::sys::thread::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}
    at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:427:23
    std::sys_common::backtrace::__rust_begin_short_backtrace
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18
 9: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17
    <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
    std::panicking::try::do_call
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40
    std::panicking::try
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19
    std::panic::catch_unwind
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14
    std::thread::Builder::spawn_unchecked_::{{closure}}
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30
    core::ops::function::FnOnce::call_once{{vtable.shim}}
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5
 10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9
    <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9      std::sys::unix::thread::Thread::new::thread_start
    at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17
 11: start_thread
 12: __GI___clone3"]
[location=/rust/git/checkouts/tokio-8e927faba632ed16/4394380/tokio-timer/src/wheel/mod.rs:225] [thread_name=steady-timer]
| username: 连连看db | Original post link

The time on the machine where TiDB is located has rolled back. Check if there have been any changes to NTP.

| username: DBAER | Original post link

Time has rolled back, no NTP.

| username: 小于同学 | Original post link

Time rollback

| username: dba远航 | Original post link

Issues with time inconsistency, check server time synchronization problems.

| username: zhang_2023 | Original post link

Time has rolled back.