TiKV Memory Usage Continues to Grow

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIKV内存使用持续增长

| username: 逍遥_猫

[TiDB Usage Environment] Production Environment
[TiDB Version] V6.5.1
[Encountered Issue: Problem Phenomenon and Impact] TIKV memory keeps growing

Deployed on a virtual machine v6.5.1. The memory limit for storage.block-cache.capacity is set to 45% of the allocated virtual machine memory (specific value 75G), but the memory usage keeps growing until OOM, then it starts growing again and OOMs again.

The official documentation has TiKV 内存参数性能调优 | PingCAP 文档中心


How to understand the sentence in the red box? If there are large queries, isn’t the space occupied by rocksdb.defaultcf.block-cache-size set? Is it occupying another memory space of rocksdb.defaultcf.block-cache-size?

| username: 像风一样的男子 | Original post link

Literally, it means caching query data into memory. Check in the dashboard under topsql to see which SQL is using so many resources, optimize the SQL or limit the SQL memory usage.

| username: 逍遥_猫 | Original post link

When there is a storage.block-cache.capacity = 75G limit, it should start clearing some cache when it reaches around 75G, right? Why does it exceed the limit?

| username: tidb菜鸟一只 | Original post link

How many GB of memory does your virtual machine have in total? Additionally, are there other components deployed on it, or is there only one TiKV?

| username: 逍遥_猫 | Original post link

Virtual memory 170G, only deployed TIKV

| username: 大飞哥online | Original post link

Single-instance TiKV or multi-instance?

| username: tidb菜鸟一只 | Original post link

That shouldn’t be the case. The storage.block-cache.capacity parameter is set to limit TiDB’s block-cache to only use 75G. Other memory usage, including the content you copied in the red box, should at most use 2/3 of the block-cache. The reason TiDB asks you to set storage.block-cache.capacity to 45% of the total server memory is to ensure that TiKV’s total memory usage does not exceed 75% of the system’s total memory, leaving 25% for the system to use. Can you take a screenshot of the host’s memory usage rate? Let’s see how much memory TiKV is actually using.

| username: chenhanneu | Original post link

If you have 256GB of memory and 4 TiKV instances, how much storage.block-cache.capacity would you recommend allocating?

| username: 路在何chu | Original post link

I understand this. Sorting and grouping require temporary memory, which means system memory will be used.

| username: tidb菜鸟一只 | Original post link

256/4*45%=28G. If it’s a multi-instance TiKV, follow what I mentioned earlier: first reserve 25% for the system, then divide the remaining 75% among the 4 instances. Each instance’s 3/5 portion will be the size of the storage.block-cache.capacity parameter.

| username: chenhanneu | Original post link

Got it, thank you.
256x0.75/4/5x3=28.8

| username: 逍遥_猫 | Original post link

Single instance