Does TiKV have a parameter similar to TiDB’s tidb_server_memory_limit_gc_trigger? Does TiKV’s memory usage keep growing? Is there a parameter similar to TiDB’s tidb_server_memory_limit_gc_trigger?
memory-usage-limit: This parameter is used to control the memory limit of the TiKV instance. You can set an appropriate value based on the actual situation to ensure that TiKV does not consume too much memory resources. The value of this parameter should be configured according to the size of the block cache[1].
block-cache-size: This parameter is used to set the block cache size of the TiKV instance. The block cache is a memory area used to cache data blocks, which can improve read performance. You can set an appropriate value based on the actual situation to balance memory usage and read performance[2].
In addition to the above parameters, you can further optimize TiKV’s memory usage through other configuration parameters, such as:
rocksdb.max-open-files: This parameter is used to set the maximum number of open files for RocksDB. If TiKV encounters the error “the maximum number of open file descriptors is too small” in some Kubernetes clusters, you can try increasing the value of this parameter[2].
Question 1: Firstly, in most cases, TiKV’s memory will not grow indefinitely. It will use memory based on the configured usage amount. There is a scenario where, when you read a large amount of data and the network returning to TiDB is slow, it will cause data accumulation. At this time, TiKV’s memory usage will spike, making it prone to OOM (Out of Memory).
Question 2: The commonly used configuration parameters related to TiKV memory are as listed above. For specific content and explanations, please refer to the official documentation.
Setting the block-cache-size to 40% of physical memory, the value of memory-usage-limit changes with the value of block-cache-size, but the actual memory usage of TiKV keeps growing until OOM, and then continues to grow. From the Grafana monitoring, it can be seen that the usage of block-cache-size is always within the specified range. I am very puzzled, what exactly is causing the TiKV memory usage to keep growing?
To set TiKV memory, you only need to configure block-cache-size. According to my tests, the maximum memory usage of TiKV is 1.25 times the block-cache-size.
SHOW config WHERE NAME LIKE ‘%block-cache.capacity%’
It shouldn’t be. After setting the block-cache.capacity size in TiKV, the memory usage of the TiKV process should not exceed 2.25 times the block-cache.capacity parameter. Could you provide the abnormal monitoring graph of TiKV at that time?