Tikv-server 6.5.0 Memory Continues to Grow: block_cache Limited to 5GB, memory_usage_limit Set to 10GB, Still Not Stopping

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv-server 6.5.0 内存持续增长 block_cache 限制5GB,memory_usage_limit 设置10GB,仍然没有停止

| username: TiDBer_TWbAeauQ

【TiDB Usage Environment】Production Environment
【TiDB Version】
【Reproduction Path】Continuous read and write, no operations
【Encountered Problem: Phenomenon and Impact】
tikv-server memory keeps growing, component triggers OOM
【Resource Configuration】
pd, tikv server mixed deployment, 3*[48 core 128GB memory 3.75TB *1]
【Attachments: Screenshots/Logs/Monitoring】




| username: TiDBer_TWbAeauQ | Original post link

Currently, TiKV 6.5.0 performs well and is relatively stable in other aspects, except that memory usage continues to increase. We hope to get professional R&D colleagues to investigate and resolve this issue as soon as possible.

| username: xfworld | Original post link

Mixed deployment? How is it mixed?

| username: TiDBer_TWbAeauQ | Original post link

PD and TiKV-server mixed deployment: Three machines, each with one PD and one TiKV-server.

| username: TiDBer_TWbAeauQ | Original post link

There might be a possibility of a memory leak. Limiting the memory_usage_limit doesn’t help, and the combined memory usage from memory_trace and block cache doesn’t match the high memory usage shown in the monitoring. All three machines have THP disabled, but the memory usage continues to grow.

| username: TiDBer_TWbAeauQ | Original post link

To add, there are no giant regions [tp99 - over 200MB], most regions are hibernated, and each server has only 50 to 60 active regions, which is in line with our expectations. We have implemented hash partitioning restrictions, so heartbeats are not a factor affecting memory. The read and write operations follow tidal traffic patterns, with peaks and troughs.

| username: TiDBer_TWbAeauQ | Original post link

Continuous growth has already reached the highest usage of 9.8GB. If there is a continuous leak, no amount of memory will be enough…

| username: xfworld | Original post link

Are there no TiDB nodes?

| username: xfworld | Original post link

It’s probably caused by no GC. Just set up an instance of a TiDB node… :rofl:

| username: TiDBer_TWbAeauQ | Original post link

Are you referring to the GC of MVCC? Where would this cause a memory spike? We use TiKV without TiDB because setting up TiDB has cost issues.

| username: TiDBer_TWbAeauQ | Original post link

It currently appears to have reached 10GB. The memory_usage_limit (set to 10GB) parameter seems to be ineffective.

| username: TiDBer_TWbAeauQ | Original post link

May I ask where this gc refers to? I’ll check the code to see if there’s a risk of not triggering garbage collection.

| username: TiDBer_TWbAeauQ | Original post link

Memory usage keeps increasing. I’ll check the mem_profile to see what exactly is growing continuously. It seems we can confirm it’s a memory leak.

| username: xfworld | Original post link

Yes, you can manually control the MVCC’s GC.

This is a feature of RocksDB.

| username: TiDBer_TWbAeauQ | Original post link

Oh, we have worked with RocksDB before, and there are many RocksDB products in the current network. TiKV’s MVCC is based on RocksDB, right? Default, write, lock, and raft CF, while RocksDB internally writes multiple keys, which has nothing to do with memory, right?

| username: TiDBer_TWbAeauQ | Original post link

To add restrictions, limit raft_engine.memory_limit to 1GB, server.grpc-memory-pool-quota to 2GB, block_cache_capacity to 4GB, and memory_usage_limit to 8GB. Theoretically, the maximum memory should be 11GB (1+2+8). If the cluster exceeds this value in a silent state and the memory continues to increase with writes, it can be confirmed that there is a memory leak.

| username: TiDBer_TWbAeauQ | Original post link

Just executed a compact on the entire cluster, but the memory hasn’t decreased at all. This method doesn’t seem to work. However, the write CF compact got stuck and hasn’t returned, probably because the write column family is very large.

| username: TiDBer_TWbAeauQ | Original post link

Moreover, after triggering the compaction of the write column, the memory usage will actually spike.

| username: TiDBer_jYQINSnf | Original post link

I’m not quite sure how the memory-usage-limit in version 6.5 is implemented. Have you checked if the memtable for RocksDB is set? The memtable also consumes a significant amount of memory. There are two sets of RocksDB: rocksdb-raft and rocksdb-kv. Are both of them configured?

The two major memory consumers in RocksDB are: blockcache and memtable.
Check if these two are useful.

| username: TiDBer_TWbAeauQ | Original post link

There is not much monitoring for memtable. In version 6.5.0, the default for raft is not rocksdb, but raft-engine 0.3 (this part of the memory is not visible). The memory monitoring for rocksdb_kv is visible, with block-cache stable at 4GB and memtable relatively small.