TiDB server restarts continuously at intervals of 5, 10, and 15 minutes

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb server间隔5,10,15分钟不断重启

| username: TiDB_New_People

To improve efficiency, please provide the following information. A clear problem description can help solve the issue faster:
【TiDB Usage Environment】Production environment, TiDB 6.1
Not sure why this statement is executed every 5, 10, 15 minutes.
select HIGH_PRIORITY table_id, is_index, hist_id, count, repeats, lower_bound, upper_bound, ndv from mysql.stats_buckets order by table_id, is_index, hist_id, bucket_id;
【Overview】
By observing slow query statements: it is found that the following statement is executed every 5 minutes, sometimes 10 minutes, 15 minutes.
select HIGH_PRIORITY table_id, is_index, hist_id, count, repeats, lower_bound, upper_bound, ndv from mysql.stats_buckets order by table_id, is_index, hist_id, bucket_id;
By observing the Linux system log, TiDB restarts the tidb 4000 service every 5 minutes
Aug 18 14:12:00 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:12:00 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:12:00 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:12:16 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:12:16 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:12:16 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:12:16 TIDB-PD1 bash: [2022/08/18 14:12:16.250 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:14:12 TIDB-PD1 kernel: audit: audit_lost=5414193 audit_rate_limit=512 audit_backlog_limit=16384
Aug 18 14:16:51 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:16:51 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:16:51 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:17:06 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:17:06 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:17:06 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:17:06 TIDB-PD1 bash: [2022/08/18 14:17:06.498 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:21:51 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:21:51 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:21:51 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:22:06 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:22:06 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:22:06 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:22:06 TIDB-PD1 bash: [2022/08/18 14:22:06.755 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:24:14 TIDB-PD1 kernel: audit: audit_lost=5414540 audit_rate_limit=512 audit_backlog_limit=16384
Aug 18 14:27:01 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:27:01 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:27:01 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:27:16 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:27:16 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:27:16 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:27:16 TIDB-PD1 bash: [2022/08/18 14:27:16.754 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:31:52 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:31:52 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:31:52 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:32:07 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:32:07 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:32:07 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:32:08 TIDB-PD1 bash: [2022/08/18 14:32:08.003 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:34:16 TIDB-PD1 kernel: audit: audit_lost=5414736 audit_rate_limit=512 audit_backlog_limit=16384
Aug 18 14:36:50 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:36:50 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:36:50 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:37:05 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:37:05 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:37:05 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:37:05 TIDB-PD1 bash: [2022/08/18 14:37:05.504 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:44:08 TIDB-PD1 systemd-logind: New session 6589 of user root.
Aug 18 14:44:08 TIDB-PD1 systemd: Started Session 6589 of user root.
Aug 18 14:44:09 TIDB-PD1 systemd-logind: New session 6590 of user root.
Aug 18 14:44:09 TIDB-PD1 systemd: Started Session 6590 of user root.
Aug 18 14:50:03 TIDB-PD1 systemd-logind: New session 6591 of user root.
Aug 18 14:50:03 TIDB-PD1 systemd: Started Session 6591 of user root.
Aug 18 14:50:03 TIDB-PD1 systemd-logind: New session 6592 of user root.
Aug 18 14:50:03 TIDB-PD1 systemd: Started Session 6592 of user root.
Aug 18 14:51:46 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 14:51:46 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 14:51:46 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 14:52:01 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 14:52:01 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 14:52:01 TIDB-PD1 systemd: Started tidb service.
Aug 18 14:52:01 TIDB-PD1 bash: [2022/08/18 14:52:01.499 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 14:54:19 TIDB-PD1 kernel: audit: audit_lost=5414925 audit_rate_limit=512 audit_backlog_limit=16384
Aug 18 15:01:01 TIDB-PD1 systemd: Started Session 6593 of user root.
Aug 18 15:01:55 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 15:01:55 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 15:01:55 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 15:02:10 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 15:02:10 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 15:02:10 TIDB-PD1 systemd: Started tidb service.
Aug 18 15:02:10 TIDB-PD1 bash: [2022/08/18 15:02:10.248 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 15:02:16 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 15:02:16 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 15:02:16 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 15:02:31 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 15:02:31 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 15:02:31 TIDB-PD1 systemd: Started tidb service.
Aug 18 15:02:31 TIDB-PD1 bash: [2022/08/18 15:02:31.501 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 15:04:21 TIDB-PD1 kernel: audit: audit_lost=5415368 audit_rate_limit=512 audit_backlog_limit=16384
Aug 18 15:06:48 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 15:06:48 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 15:06:48 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 15:07:03 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 15:07:03 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 15:07:03 TIDB-PD1 systemd: Started tidb service.
Aug 18 15:07:03 TIDB-PD1 bash: [2022/08/18 15:07:03.503 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 15:07:08 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 15:07:08 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 15:07:08 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 15:07:23 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 15:07:23 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 15:07:23 TIDB-PD1 systemd: Started tidb service.
Aug 18 15:07:24 TIDB-PD1 bash: [2022/08/18 15:07:24.001 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 15:11:46 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 15:11:46 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 15:11:46 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 15:12:01 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 15:12:01 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 15:12:01 TIDB-PD1 systemd: Started tidb service.
Aug 18 15:12:01 TIDB-PD1 bash: [2022/08/18 15:12:01.754 +08:00] [WARN] [config.go:1004] [“Some configuration options should be moved to [instance] section. Please use the latter config options in [instance] instead: (slow-threshold, tidb_slow_log_threshold).”]
Aug 18 15:14:23 TIDB-PD1 kernel: audit: audit_lost=5415762 audit_rate_limit=512 audit_backlog_limit=16384
Aug 18 15:16:48 TIDB-PD1 systemd: tidb-4000.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 15:16:48 TIDB-PD1 systemd: Unit tidb-4000.service entered failed state.
Aug 18 15:16:48 TIDB-PD1 systemd: tidb-4000.service failed.
Aug 18 15:17:03 TIDB-PD1 systemd: tidb-4000.service holdoff time over, scheduling restart.
Aug 18 15:17:03 TIDB-PD1 systemd: Stopped tidb service.
Aug 18 15:17:03 TIDB-PD1 systemd: Started tidb service.
Aug 18 15:17:03 TIDB-PD1 bash: [2022/08/18 15:17:03.504 +08:00] [WARN] [config.go:1004] ["Some configuration options should be moved to [instance] section. Please use the

2
| username: TiDB_New_People | Original post link

The image you provided is not accessible. Please provide the text content you need translated.

3
| username: db_user | Original post link

Could you please provide the edit-config information and the show config information?

4
| username: TiDB_New_People | Original post link

tidb_stderr.logtidb_stderr.log (10.4 MB)

5
| username: jansu-dev | Original post link

pingcap/tidb/expression/constant.go:91 → Points to a prepare issue;
However, it is not yet known if the prepare issue is causing the restart problem. You can first resolve the prepare issue (as mentioned in the previous post’s question…).
The probability is high.