Memory Control Issues in TiKV Process

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv进程内存控制问题

| username: 月明星稀

Dear experts, if I want to control the memory usage of tikv-server to 5G, which configurations should I adjust and what are the recommended values? Please kindly advise.

| username: Jellybean | Original post link

What is your scenario for setting TiKV’s memory so low?

You can control the memory usage of the storage layer to some extent by adjusting the block cache.

| username: 月明星稀 | Original post link

The machine has relatively little memory and doesn’t store much data, but there are quite a few concurrent requests.

| username: 月明星稀 | Original post link

This is configured with 500M, but the memory usage keeps increasing.

| username: Jellybean | Original post link

Well, for memory usage, you also need to set related parameters for RocksDB, such as controlling the memory usage size of default CF, Write CF, and Lock CF.

Moreover, in actual use, it will also be related to the amount of data read and written, so you can’t set it too low.

The official documentation has a minimum memory usage requirement, which I remember is larger than 5GB. You can check the “Deployment Resource Requirements” section.

| username: 月明星稀 | Original post link

I’ve already set the default CF. Will the write CF and lock CF consume a lot of memory?

| username: 春风十里 | Original post link

The production requires TiKV memory to be above 64GB.

| username: Kongdom | Original post link

Even in a testing environment, TiKV requires 32GB of memory.

Can we refer to the hybrid deployment tuning here and adjust the thread control to optimize memory usage?

| username: Miracle | Original post link

K8S can perfectly meet your requirements :smirk:

| username: 江湖故人 | Original post link

TiKV Memory Parameter Performance Tuning | PingCAP Documentation Center

In addition to the block-cache and write-buffer listed above that will occupy system memory:

  1. Some memory needs to be reserved as the system’s page cache.
  2. When TiKV processes large queries (such as select * from ...), it will read data and then generate the corresponding data structure in memory to return to TiDB. During this process, TiKV will occupy a portion of the memory.
| username: 江湖故人 | Original post link

Containers and virtualization technologies provide more thorough isolation. :+1:

| username: liyuepeng123 | Original post link

This article introduces the hybrid deployment topology of TiKV and TiDB in a TiDB cluster and the main parameters. A common scenario is that the deployment machine has multi-way CPU processors and sufficient memory. To improve the utilization of physical machine resources, multi-instance deployment on a single machine can be used, i.e., TiDB and TiKV are bound to cores through NUMA, isolating CPU resources. PD and Prometheus are deployed together, but their data directories need to use independent file systems.

| username: 有猫万事足 | Original post link

Do not set each cf separately.
You can set the total amount through storage.block-cache.capacity. I set my own TiKV to around 2GB, which perfectly controls the total memory usage of the entire TiKV to not exceed 6GB.

| username: h5n1 | Original post link

Could you post a graph showing the memory growth trend of TiKV? The TiKV detail monitoring has a memory trace; first, check how large the components can be traced.

| username: dba远航 | Original post link

Set it through storage.block-cache.capacity.

| username: tidb菜鸟一只 | Original post link

Based on your resource allocation, I suggest not using TiDB. Using a single MySQL instance would be more efficient.

| username: Kongdom | Original post link

:thinking: I second that, it is indeed the case. However, if it’s for compatibility verification, setting up a single node might provide a better experience.

| username: TiDBer_小阿飞 | Original post link

I second that, without the need for HTAP, a single machine would be better.

| username: 路在何chu | Original post link

Well, I also set this up, and the memory control is quite good.

| username: TiDBer_jYQINSnf | Original post link

Try the following TiKV configuration and be sure to reply with the results. I also want to know if it can run with 5G, including continuous writes, and whether WAL recovery after a restart can cause OOM issues. :smile:

memory-usage-limit = "4G"

[server]
grpc-memory-pool-quota = "1G"
[storage.block-cache]
capacity = "1GB"
[rocksdb]
max-total-wal-size = "2GB"
enable-statistics = false
[rocksdb.defaultcf]
write-buffer-size = "64MB"
max-write-buffer-number = 2
[rocksdb.writecf]
write-buffer-size = "64MB"
max-write-buffer-number = 2
[rocksdb.lockcf]
write-buffer-size = "64MB"
max-write-buffer-number = 2
[raftdb.defaultcf]
write-buffer-size = "64MB"
max-write-buffer-number = 2
[raft-engine]
memory-limit = "512MB"