TiDB-server memory usage is 200G, and even after there are no connections in the processlist, it still cannot perform normal GC on the memory

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb-server内存占用200G,processlist无连接后,还不能正常GC内存。

| username: ycong

【TiDB Usage Environment】Production Environment
【TiDB Version】v7.5.0
【Encountered Issue: Symptoms and Impact】
During normal usage, after enabling Alertmanager, an alert was received indicating that the memory usage has been at 200G for a long time. The TiDB-server flame graph shows that ParseSQL alone occupies over 50GB.
Setting tidb_server_memory_limit=150G did not reduce the memory usage.
Subsequently, all traffic to this node was pulled, and the processlist was cleared, but the memory usage still did not decrease.

【Attachments: Screenshots/Logs/Monitoring】


| username: WalterWj | Original post link

Changed to product defect feedback. It feels like a memory overflow that hasn’t been released… If it’s urgent, I think you can try restarting… :thinking:

| username: DBRE | Original post link

Version 7.5 still has this issue?

| username: ycong | Original post link

Memory restart can solve the issue. We think this scenario is quite valuable, so we have preserved the scene and haven’t restarted yet.

| username: ycong | Original post link

[2024/05/31 11:48:43.102 +08:00] [WARN] [servermemorylimit.go:179] [“Global memory controller tries to kill the top memory consumer, but no one larger than tidb_server_memory_limit_sess_min_size is found”] [tidb_server_memory_limit_sess_min_size=134217728]

| username: ycong | Original post link

This TiDB node currently has no business, but the CPU is still very high, and the CPU consumption is also in the GC thread.

Created a scenario (first set tidb_server_memory_limit_sess_min_size to be small enough, then execute a large query) to let the code execute to runtime.gc, but the memory was not released.

| username: tidb菜鸟一只 | Original post link

Are TiFlash and TiDB placed together? Have you done NUMA isolation?

| username: hawkingrei | Original post link

It is recommended to directly disable tidb_enable_historical_stats. This feature currently has issues and has already been disabled by default on the master branch.