TiFlash fails to start after going down

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiflash down掉之后无法启动

| username: Hacker_suny

【TiDB Environment】Production environment
【TiDB Version】4.0.11
【Problem Encountered】
At midnight today, TiFlash suddenly went down and could not be started.
【Reproduction Path】None
【Problem Phenomenon and Impact】
Error keywords are as follows:
[2022/09/28 10:47:53.846 +08:00] [ERROR] [] [“Application: Bootstrap failed because sync schema error: Syntax error: Not a valid integer: 0.00
We will sleep for 3 seconds and try again.”] [thread_id=1]

The logs show a large amount of synced information:
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).users_to_role(45954), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).users_to_role(45954) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_201904(45956) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_201904(45956) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_201904(45956)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_201904(45956), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_201904(45956) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_201905(45958) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_201905(45958) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_201905(45958)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_201905(45958), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_201905(45958) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).channel_lines_limit(45960) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).channel_lines_limit(45960) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).channel_lines_limit(45960)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).channel_lines_limit(45960), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).channel_lines_limit(45960) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).tmp_buxiaofan_220206(45965) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).tmp_buxiaofan_220206(45965) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).tmp_buxiaofan_220206(45965)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).tmp_buxiaofan_220206(45965), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).tmp_buxiaofan_220206(45965) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202202(46189) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202202(46189) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202202(46189)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202202(46189), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202202(46189) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202203(54001) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202203(54001) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202203(54001)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202203(54001), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202203(54001) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202204(55433) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202204(55433) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202204(55433)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202204(55433), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202204(55433) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202205(62487) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202205(62487) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202205(62487)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202205(62487), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202205(62487) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202206(63616) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202206(63616) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202206(63616)”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202206(63616), not altering”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202206(63616) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202207(68398) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202207(68398) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.061 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202207(68398)”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202207(68398), not altering”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202207(68398) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202208(73754) syncing during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202208(73754) name identical, not renaming.”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [INFO] [] [“SchemaBuilder: Altering table qrp(45756).trade_jnls_202208(73754)”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [INFO] [] [“SchemaBuilder: No schema change detected for table qrp(45756).trade_jnls_202208(73754), not altering”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [DEBUG] [] [“SchemaBuilder: Table qrp(45756).trade_jnls_202208(73754) synced during sync all schemas”] [thread_id=1]
[2022/09/28 10:52:15.062 +08:00] [DEBUG] [] [“SchemaBuilder: Database qrp(45756) synced during sync all schemas”] [thread_id=1]

【Attachment】Related logs and monitoring (https://metricstool.pingcap.com/)

| username: h5n1 | Original post link

Refer to this: The default value of the int type column is set to 0.00, and the final rebuilt table

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.