TiKV Memory Configuration Parameters

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv内存配置参数

| username: liyuntang

【TiDB Usage Environment】Production Environment
【TiDB Version】7.1.0
【Reproduction Path】None
【Encountered Issues: Problem Phenomenon and Impact】
【Resource Configuration】
【TiDB Operator Version】: v1.4.6
【K8s Version】: v1.19
【Attachments: Screenshots/Logs/Monitoring】

memory-usage-limit
storage.block-cache.capacity
raftstore.capacity
raft-engine.memory-limit

In the TiKV configuration parameters, there are several memory-related configuration parameters as listed above. The questions are as follows:

  1. What are the impacts of these parameters respectively? If the configuration is unreasonable (e.g., pod memory is 10g, and the parameter configuration exceeds 10g), what will be the effect?
  2. Is there any intrinsic relationship between these parameters, for example, their sum should not exceed the pod memory?
  3. In a pod with 64g memory, what proportion is recommended for configuring these parameters, or can you directly provide a reference configuration?
| username: tidb菜鸟一只 | Original post link

You only need to set storage.block-cache.capacity to 45% of your pod’s memory. For example, if the pod has 10GB of memory, set the parameter to 4.5GB. No other settings are required.

| username: liyuntang | Original post link

If these parameters are not set, will using the default values cause the pod to OOM?

| username: liyuntang | Original post link

Especially for parameters like memory-usage-limit, it feels similar to MySQL’s buffer pool. If the size is not explicitly set, once multiple TiKV instances exist on the same physical machine, it will definitely trigger OOM.

| username: Mark | Original post link

It is better to reserve 15-20% of the memory, as the load pressure during peak periods may be too much to handle in time. :joy:

| username: tidb菜鸟一只 | Original post link

You only need to set storage.block-cache.capacity. After setting this, the memory usage of a single TiKV will not exceed 2.2 times the value of storage.block-cache.capacity. This is similar to MySQL’s buffer pool used for caching data.

The memory-usage-limit has a general effect on limiting TiKV’s memory. This setting limits the total memory usage of TiKV, but sometimes it may not be effective.

If you have a 64G machine with one TiKV, set the memory-usage-limit to 64G * 0.45.
If you have two TiKVs, set it to 64G / 2 * 0.45, and so on. Setting it this way will ensure that your TiKV will not trigger an out-of-memory (OOM) error.

| username: liyuntang | Original post link

Okay, understood.

| username: liyuntang | Original post link

If the raft-engine.memory-limit parameter is set too high, will it cause the pod to OOM?

| username: zhanggame1 | Original post link

Just set storage.block-cache.capacity for TiKV, don’t touch anything else.

| username: YuchongXU | Original post link

Reserve a bit more, just in case.

| username: liyuntang | Original post link

Our TiDB cluster is running in a Kubernetes environment, and we have many other TiDB clusters as well. If we don’t configure it, will there be a risk of OOM (Out of Memory) or performance issues?

| username: liyuntang | Original post link

“If the memory usage of a single TiKV will not exceed 2.2 times the value of storage.block-cache.capacity, then according to this formula, if a pod is allocated 22GB of memory, the maximum value for storage.block-cache.capacity should be set to 10GB. Is this correct?”

| username: liyuntang | Original post link

Actually, my requirement here is to ensure that the total memory usage of TiKV does not exceed 80% of the pod’s memory. For example, if the pod’s memory is 100GB, then the entire TiKV program’s memory usage should not exceed 80GB.

| username: tidb菜鸟一只 | Original post link

Yes, if you configure it this way, your pod will not experience OOM. The 2.2 times already reserves some memory for the operating system.

| username: liyuntang | Original post link

Okay, understood.