TiFlash fails to restart after an error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash 报错后无法重启

| username: Hacker_ojLJ8Ndr

【 TiDB Version】
6.5.2

tiflash_error.log

[2023/09/04 10:11:31.005 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:11:53.520 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:12:15.817 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:12:38.040 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]

tiflash_stderr.log

Logging debug to /data01/deploy/log/tiflash.log
Logging errors to /data01/deploy/log/tiflash_error.log
deprecated configuration, log-file has been moved to log.file.filename
override log.file.filename with log-file, "/data01/deploy/log/tiflash_tikv.log"

tiflash_tikv.log

[2023/09/04 08:36:32.760 +08:00] [ERROR] [observer.rs:290] ["transfer leader won't exec"] [req="cmd_type: TransferLeader transfer_leader { peer { id: 4900743611 store_id: 1151850759 } }"][region="id: 94768718 start_key: 7480000000000079FF675F72800000000AFFCDECB70000000000FA end_key: 7480000000000079FF6800000000000000F8 region_epoch { conf_ver: 236240 version: 7351 } peers { id: 4900743611 store_id: 1151850759 } peers { id: 4900826256 store_id: 4453837499 } peers { id: 4900875948 store_id: 3755910046 } peers { id: 4901151587 store_id: 4858181679 role: Learner } peers { id: 4901153408 store_id: 4745761616 role: Learner }"]

tiflash.log (repeatedly reporting errors and restarting)

[2023/09/04 10:14:09.454 +08:00] [DEBUG] [Server.cpp:995] ["Destroyed global context."] [thread_id=1]
[2023/09/04 10:14:09.454 +08:00] [INFO] [Server.cpp:868] ["Let tiflash proxy shutdown"] [thread_id=1]
[2023/09/04 10:14:09.454 +08:00] [INFO] [Server.cpp:871] ["Wait for tiflash proxy thread to join"] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [INFO] [Server.cpp:873] ["tiflash proxy thread is joined"] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [INFO] [<unknown>] ["shutting down"] [source=Application] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [DEBUG] [<unknown>] ["Uninitializing subsystem: Logging Subsystem"] [source=Application] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [INFO] [BaseDaemon.cpp:315] ["Stop SignalListener thread"] [source=BaseDaemon] [thread_id=67]
[2023/09/04 10:14:24.956 +08:00] [INFO] [<unknown>] ["Welcome to TiFlash"] [thread_id=1]
| username: ShawnYan | Original post link

Similar to this

| username: Hacker_ojLJ8Ndr | Original post link

It seems different. The continuous analysis feature is turned off. It is set to two replicas, and currently, it is version 6.2. For now, this issue has been addressed by re-expanding.

| username: Hacker_ojLJ8Ndr | Original post link

However, the log reports this: Tombstoned table db_45.t_68737(68737), but it’s only at INFO level.

| username: redgame | Original post link

Restart the TiFlash process and then check the TiFlash log files to see if there are any other error messages.

| username: Hacker_ojLJ8Ndr | Original post link

I didn’t see any other error messages; the logs are still reporting the same issue.

| username: Hacker_ojLJ8Ndr | Original post link

It might be because of this reason, the issue occurred while running a job, TiFlash经常触发OOM,重启依然无法阻止OOM发生 - TiDB 的问答社区

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.