Will installing security monitoring on ECS cause TiDB to run out of memory (OOM)?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 大佬们问一个问题 ecs里面装了安全监控 会导致tidb oom吗

| username: tidb狂热爱好者

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 7.1.5
[Reproduction Path] What operations were performed that caused the issue
[Encountered Issue: Problem Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]
The default host has 500MB of memory consumed by security monitoring software. Should we specify the memory size for each PD, TiDB, and TiKV instead of letting it automatically allocate?

2
| username: lemonade010 | Original post link

There is a default memory limit.

tidb_server_memory_limit introduced from version v6.4.0

  • Scope: GLOBAL
  • Persisted to the cluster: Yes
  • Controlled by Hint SET_VAR: No
  • Default value: 80%
  • Value range:
    • You can set this variable as a percentage, indicating the percentage of total memory usage, with a range of [1%, 99%].
    • You can also set the variable as a memory size, with a range of 0 and [536870912, 9223372036854775807], in bytes. It supports memory formats with units “KiB|MiB|GiB|TiB”. A value of 0 means no memory limit.
    • If the set memory value is less than 512 MiB and not 0, TiDB will use 512 MiB as a substitute.
  • This variable specifies the memory limit for the TiDB instance. When memory usage reaches this limit, TiDB will cancel the SQL statement with the highest current memory usage. After successfully canceling the SQL statement, TiDB will attempt to call Golang GC to immediately reclaim memory to alleviate memory pressure as quickly as possible.
  • Only SQL statements with memory usage greater than tidb_server_memory_limit_sess_min_size will be prioritized for cancellation.
  • Currently, TiDB can only cancel one SQL statement at a time. If TiDB completely cancels one SQL statement and reclaims resources, but memory usage is still greater than the limit set by this variable, TiDB will start the next cancellation operation.
3
| username: zhaokede | Original post link

I haven’t encountered that.

4
| username: 啦啦啦啦啦 | Original post link

Don’t the experts all use machines with 1TB of memory? 500MB can be ignored, right?

5
| username: TiDBer_QYr0vohO | Original post link

TiDB has a default memory limit, meaning if your machine has 16GB, you can allocate 80% for TiDB usage, and the remaining 20% is reserved for other processes or services.

6
| username: 像风一样的男子 | Original post link

500MB of memory isn’t much, right? Doesn’t the operating system usually reserve one or two gigabytes of memory by default?

7
| username: tidb狂热爱好者 | Original post link

Your answer is reliable.

8
| username: 友利奈绪 | Original post link

Is it explained in the documentation, boss?

9
| username: 友利奈绪 | Original post link

Learning firsthand

10
| username: tidb菜鸟一只 | Original post link

By default, about 20% is reserved for the system, otherwise the system would crash directly…

11
| username: TiDBer_QYr0vohO | Original post link

Yes, by default, 80% of the machine’s memory is used as the threshold for the tidb-server instance, and in practice, it will be slightly higher.