TiDB Server Node GC-Threshold Surge

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb server节点gc-threshold飙升

| username: 特雷西-迈克-格雷迪

【TiDB Usage Environment】Production Environment / Testing / PoC
【TiDB Version】
【Reproduction Path】
【Encountered Problem: Problem Phenomenon and Impact】
【Resource Configuration】“Enter tidb’s grafana: ienjoys-tidb-prod / ienjoys-tidb-prod-TiDB-Runtime”

How can I solve this problem? Why does it cause a memory spike, and the limit setting (export GOMEMLIMIT=14000MiB) doesn’t work?

【Attachments: Screenshots / Logs / Monitoring】


config.txt (3.8 KB)

| username: xfworld | Original post link

I’m quite curious, in what scenarios can the memory be fully utilized…

| username: 特雷西-迈克-格雷迪 | Original post link

There are report queries with relatively large intermediate result sets.

| username: Billmay表妹 | Original post link

Check the top SQL.

| username: ffeenn | Original post link

If it’s caused by SQL, the amount of SQL is quite large. With 40GB of memory, the stack must have overflowed.

| username: zhang_2023 | Original post link

The processes on the system itself cannot be restricted in terms of memory usage.

| username: xfworld | Original post link

Please add the monitoring information you have tracked. It is basically determined to be caused by slow queries…

| username: WalterWj | Original post link

  1. First, identify the SQL statements causing high memory usage.
  2. Provide the execution plans of the relevant SQL statements and optimize them accordingly.
| username: crazycs520-PingCAP | Original post link

TiDB has a server-memory-quota that can be used to limit the memory usage of the entire TiDB server, which should be useful for your scenario.

| username: 特雷西-迈克-格雷迪 | Original post link

OK, thanks.

| username: Soysauce520 | Original post link

Upgrading to 6.5 will kill it. The old version has issues with memory limits, and some SQL queries can’t be restricted.

| username: 友利奈绪 | Original post link

Upgrade to a higher version, and then set the statement timeout to kill and optimize the slow statements.

| username: 小于同学 | Original post link

There are report queries with relatively large intermediate result sets.