Is it normal for tikv-server to use 20.4GB of memory with 110 million records?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 1.1亿条数据, tikv-server内存占 20.4g, 正常吗

| username: TB77506668

【TiDB Environment】Testing
【TiDB Version】6.4.0
【Reproduction Path】110 million rows of data, tikv-server memory usage is 20.4g, is this normal?
【Encountered Problem: Phenomenon and Impact】
【Resource Configuration】32 cores, 48g memory
【Attachment: Screenshot/Log/Monitoring】

image
image907×528 23.7 KB

2
| username: tidb菜鸟一只 | Original post link

That depends on how long you’ve been running it. Over 100 million rows of data, 20GB is considered normal.

3
| username: TB77506668 | Original post link

There are only 5 fields, the data shouldn’t be entirely written to memory.

4
| username: buddyyuan | Original post link

You can check how much storage.block-cache.capacity is set to.

5
| username: 逍遥_猫 | Original post link

It feels like the data extracted by this statement is concentrated on this TiKV node. I suggest checking the hotspot data and possibly splitting it.

6
| username: 裤衩儿飞上天 | Original post link

The database has 110 million records, and the memory usage of tikv-server is not necessarily related. No matter how much data there is, if it is not used, it will not consume much of your memory and will just exist on the disk. If the memory usage of a single tikv is high, you need to check if the data is unbalanced.

7
| username: TB77506668 | Original post link

This is the default, it seems to be 1G.

[storage.block-cache]

Whether to create a shared block cache for all CFs of RocksDB.

RocksDB uses block cache to cache uncompressed data blocks. A larger block cache can speed up read operations.

It is recommended to enable the shared block cache parameter. This way, you only need to set the total cache size, making the configuration process more convenient.

In most cases, the LRU algorithm can automatically balance cache usage among different CFs.

The remaining configurations in the storage.block-cache session only take effect when the shared block cache is enabled.

shared = true

The size of the shared block cache. Normally, it should be set to 30%-50% of the total system memory.

If this parameter is not set, it is determined by the sum of the following fields or their default values.

* rocksdb.defaultcf.block-cache-size or 25% of the total system memory

* rocksdb.writecf.block-cache-size or 15% of the total system memory

* rocksdb.lockcf.block-cache-size or 2% of the total system memory

* raftdb.defaultcf.block-cache-size or 2% of the total system memory

To deploy multiple TiKV nodes on a single physical machine, this parameter needs to be explicitly configured.

Otherwise, OOM errors may occur in TiKV.

capacity = “1GB”

8
| username: h5n1 | Original post link
9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.