TiDB Node Memory Surge Causes EC2 Machine to Enter Unresponsive State After Memory is Full, Ports Remain Active but Unable to Provide Service

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb节点内存异常飙升导致EC2机器在内存满之后机器假死状态但是端口还存活，但是不能提供服务

| username: Hacker_PI26psBF

【TiDB Usage Environment】Production environment
【TiDB Version】Upgraded from tidb5.4 to version 6.1
【Reproduction Path】Operations performed that led to the issue
【Encountered Issue: Phenomenon and Impact】We adjusted a parameter to tidb_mem_quota_query = 10G. After the TiDB node’s memory was exhausted, we reverted the adjustment to its initial value. However, after reverting, one TiDB node’s memory continuously increases without any slow logs, error logs, or even many normal logs. The number of connections, latency, etc., are all stable.
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】

tidb1
tidb11380×400 61.7 KB

tidb2
tidb21380×280 73.6 KB

tidb3
tidb31380×414 106 KB

tidb4
tidb41380×348 94.3 KB

tidb5
tidb51380×404 72 KB

tidb6
tidb61380×406 92.5 KB

2
| username: Billmay表妹 | Original post link

Based on your description, the issue might be due to high memory usage by the TiDB node. Here are some recommended measures:

  1. Check if the TiDB version has any known memory leak issues. If so, upgrade to the latest version.

  2. Inspect the TiDB configuration file to ensure parameters are correctly set, such as whether tidb_mem_quota_query is properly configured.

  3. Use the TiDB monitoring panel or Grafana to monitor TiDB’s memory usage. Identify which components are consuming a large amount of memory and check for any abnormal fluctuations in memory usage.

  4. Adjust TiDB parameters, such as tidb_mem_quota_query and tidb_mem_quota_query_max, to control TiDB’s memory usage.

  5. Adjust the number of TiDB connections to avoid excessive memory usage due to too many connections.

  6. Optimize query statements and plans, such as using indexes and avoiding full table scans, to reduce memory usage.

  7. If high memory usage by the TiDB node causes the machine to become unresponsive, consider restarting the TiDB node or using TiDB’s dynamic parameter adjustment feature to gradually adjust parameters and observe memory usage.

It is important to note that TiDB memory usage issues can be complex and require analysis based on specific circumstances. It is recommended to back up data before attempting to resolve the issue to prevent data loss. Additionally, when deploying TiDB, set parameters reasonably based on actual business needs and hardware configurations to avoid high memory usage and potential machine unresponsiveness.

3
| username: Billmay表妹 | Original post link

Please provide the following: Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

4
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.