A locally deployed TiDB cluster suddenly fails to start TiDB, PD, and KV components. The initial suspicion is that the inconsistency in CentOS system time is causing the startup failure

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 本地部署的tidb集群，突然出现tidb，pd，kv无法启动，初步猜测是centos的系统时间不一致导致无法启动

| username: tianjun

[TiDB Usage Environment]
Production Environment
[TiDB Version]
v7.1.0
[Reproduction Path] Operations performed that led to the issue
System restart.

[Encountered Issue: Symptoms and Impact]
TiDB, PD, and KV cannot start. Both PD and KV logs contain a “system time jump backward” error. Checking the system log shows:

[2023/10/31 16:42:22.651 +08:00] [ERROR] [systimemon.go:38] [“system time jump backward”] [last=1698741767902844769] [error=“[PD:common:ErrIncorrectSystemTime]incorrect system time”]
[2023/10/31 16:42:22.651 +08:00] [ERROR] [server.go:556] [“system time jumps backward”] [error=“[PD:common:ErrIncorrectSystemTime]incorrect system time”]
[Resource Configuration] Navigate to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Logs/Monitoring]

c923a60bd258413c5a7ee07d094565c
c923a60bd258413c5a7ee07d094565c1079×451 12.2 KB

2
| username: 像风一样的男子 | Original post link

Check the time on the operating system to see if there is any lag.

3
| username: 随缘天空 | Original post link

Before installing the TiDB cluster, you need to configure time synchronization for each node machine. Refer to the following link for checking and installing the NTP service: TiDB 环境与系统配置检查 | PingCAP 文档中心

4
| username: Jolyne | Original post link

Judging from the above error, it seems to be a time error. What system are you using, a domestic system? Check if the time zone and time are correct.

5
| username: Miracle | Original post link

Is NTP installed?

6
| username: tianjun | Original post link

CentOS 7 installed locally

7
| username: tianjun | Original post link

Installed.

8
| username: Jolyne | Original post link

Check the current time on each machine using the date command to see if they are consistent, or use ntpq -p to check the synchronization status.

9
| username: zhanggame1 | Original post link

Check with ntpq -p. Also, is it a virtual machine?

10
| username: TiDBer_小阿飞 | Original post link

Modify CPU time source parameters logs - Linux system time temporarily jumps - Unix & Linux Stack Exchange

clocksource=    Override the default clocksource
                Format: <string>
                Override the default clocksource and use the clocksource
                with the name specified.
                Some clocksource names to choose from, depending on
                the platform:
                [all] jiffies (this is the base, fallback clocksource)
                [ACPI] acpi_pm
                ...
                [X86-64] hpet,tsc

cat /sys/devices/system/clocksource/clocksource*/available_clocksource

11
| username: xingzhenxiang | Original post link

Check before installation

12
| username: Fly-bird | Original post link

Time is quite important.