TiFlash suddenly crashed and won't start

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash突然挂掉,启动不起来

| username: Zealot

[TiDB Usage Environment] Production Environment
[TiDB Version] 7.1.2
[Reproduction Path] No operations performed
[Encountered Problem: Phenomenon and Impact]
After suddenly crashing, checked the log:


Attempted access has violated the permissions assigned to the memory area
After restarting, it fails to start with the error:
[2024/06/05 16:11:18.690 +08:00] [ERROR] [] ["DB::Exception: Cannot create table from metadata file /data/deploy/install/data/tiflash-9000/metadata/db_1077654/t_1487435.sql, error: DB::Exception: Cannot parse string ‘CURRENT_DATE’ as MyDate: syntax error at position 8 (parsed just ‘CURRENT_’), stack trace:\n\n 0x1c54f3e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) [tiflash+29708094]

File, t_1487435.sql:

ATTACH TABLE t_1487435

(

id Int32,

company Nullable(String),

illegal_person Nullable(String),

original_qualifications_levels Nullable(String),

now_qualifications_levels Nullable(String),

registered_capital Nullable(String),

dataid Nullable(String),

declaration_type Nullable(String),

declaration_time Nullable(String),

annex_path Nullable(String),

annex_web_url Nullable(String),

dataid_new Nullable(String), q

data_caiji Nullable(MyDate) DEFAULT CAST(‘CURRENT_DATE’ AS Nullable(MyDate)),

apt_code Nullable(String),

apt_code_date Nullable(MyDate),

project_name Nullable(String),

project_id Nullable(String),

company_id Nullable(String),

company_name Nullable(String),

apt_name Nullable(String)

)

ENGINE = DeltaMerge(id,

After removing the folder db_1077654, it can start normally.

| username: tidb菜鸟一只 | Original post link

Find the corresponding table, then set the TiFlash replica to 0 and try again. There might be an issue with the table structure of this table.

| username: Zealot | Original post link

I set all the table’s TiFlash replicas to 0, but this error still occurs, and this table has not been added to TiFlash. Here is the original table DDL:

CREATE TABLE `zhujianbu_company_declare_qualification_info` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `declaration_time` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '申报时间',
  `company` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '企业名称',
  `company_name` varchar(255) DEFAULT NULL COMMENT '企业名称清洗',
  `company_id` varchar(50) DEFAULT NULL COMMENT '企业id',
  `apt_name` varchar(255) DEFAULT NULL COMMENT '资质名称',
  `apt_code` varchar(255) DEFAULT NULL COMMENT '资质代码-反洗数据',
  `apt_code_date` date DEFAULT NULL COMMENT '资质取得日期',
  `now_qualifications_levels` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '现资质证书',
  `project_name` varchar(255) DEFAULT NULL COMMENT '最后使用的业绩名称:排除多次申请业绩不符项',
  `project_id` varchar(50) DEFAULT NULL COMMENT '四库业绩编号',
  `illegal_person` varchar(100) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '法人',
  `original_qualifications_levels` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '原资质证书',
  `registered_capital` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册资金',
  `declaration_type` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '申报类型',
  `annex_path` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件路径',
  `annex_web_url` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件网页url',
  `dataid` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT 'dataid',
  `dataid_new` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `data_caiji` date DEFAULT CURRENT_DATE COMMENT '采集日期',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=570001
| username: WalterWj | Original post link

Did you upgrade recently? Previously it was below version 650… If so, you might have encountered a bug.
I recommend redoing TiFlash: TiFlash replica 0 >> scale down TiFlash >> scale up TiFlash >> TiFlash replica 2

| username: FutureDB | Original post link

Indeed, you can try this solution. Previously, when we used Flashback to recover accidentally deleted data (some of the data had TiFlash replicas), it triggered a TiFlash bug, causing TiFlash to fail to start. The approach we took was to first delete the TiFlash replicas, then scale in and out the TiFlash nodes, and finally reset the TiFlash replicas.

| username: 小于同学 | Original post link

Upgrade the version.

| username: tidb菜鸟一只 | Original post link

So, did you really upgrade your cluster? I remember that in version 5, TiFlash would create the TiFlash table structure even if the table didn’t have a TiFlash replica. In version 7, it should only create the table structure for tables that have TiFlash replicas. Can you check if all databases and tables have table structures in the /data/deploy/install/data/tiflash-9000/metadata/ directory, regardless of whether they have TiFlash replicas?

| username: ziptoam | Original post link

DB::Exception: Cannot parse string ‘CURRENT_DATE’ as MyDate. This indicates that TiDB encountered an issue parsing the default value expression CURRENT_DATE while attempting to restore the table structure from the metadata file t_1487435.sql. TiDB or its component TiFlash expects a valid date format here, but a syntax error occurred when directly using CURRENT_DATE as the default value.

| username: TiDBer_rvITcue9 | Original post link

Study and learn