TiFlash CPU and Memory Spike Causing Restart

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash CPU和内存突然升高导致重启

| username: a398058068

【TiDB Usage Environment】Production Environment
【TiDB Version】TIDB 6.1.2
【Encountered Problem】TiFlash auto-restarts
【Reproduction Path】What operations were performed that led to the problem
【Problem Phenomenon and Impact】
CPU and memory suddenly increased, then TiFlash restarted, but no useful information was found in the logs.

Check if there are any SQL statements hitting TiFlash using the TOP SQL and SQL statement analysis features in the dashboard. Traffic visualization shows significant traffic at that time, but it is impossible to pinpoint the specific problem or statement. The largest memory-consuming query in the slow query log is only 200MB, and the others are just a few dozen, which doesn’t seem to be the root cause.

【Attachments】Related logs and monitoring (https://metricstool.pingcap.com/)
tiflash-monitor&log.zip (63.1 KB)

For questions related to performance optimization or fault troubleshooting, please download the script and run it. Please select all and copy-paste the terminal output.

2
| username: h5n1 | Original post link

Refer to this

3
| username: a398058068 | Original post link

I have seen this before and it doesn’t seem to be an issue.
I am looking at traffic visualization from the dashboard and can see there is traffic, but I can’t find the specific SQL hitting TiFlash. It’s very strange.

4
| username: flow-PingCAP | Original post link

Sorry, metricstool.pingcap.com is no longer maintained. It is recommended to use clinic to re-upload a diagnostic report as referenced here: PingCAP Clinic 快速上手指南 | PingCAP 文档中心

5
| username: a398058068 | Original post link
6
| username: TiDBer_CEVsub | Original post link

Could it be related to hardware failure?

7
| username: a398058068 | Original post link

It might not have been described clearly, but TiFlash restarted automatically.

8
| username: JinheLin | Original post link

I suggest checking dmesg and /var/log/messages for any other error messages.

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.