High Memory Usage in TiKV

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIKV 内存使用率过高

| username: 饭光小团

[TiDB Usage Environment] Production Environment
[TiDB Version] v5.3.1
[Encountered Problem] KV node memory usage exceeds limit
[Reproduction Path] None
[Problem Phenomenon and Impact]
The online setting for .block_cache.capacity is 10G, but it is found that the running memory of TiKV is much larger than this setting. By observing the memory information, it can be seen that the KV instance has allocated many anon memory pages. I would like to ask how to limit the memory usage of KV and where this memory is roughly used.
tiup ctl:v5.3.1 tikv --host XX.XX.XXX.35:20171 modify-tikv-config -n storage.block_cache.capacity -v 10GB

image
image1380×316 67.4 KB

image
image
image1380×298 63.1 KB

cat /proc/meminfo
MemTotal: 131450928 kB
MemFree: 1358108 kB
MemAvailable: 14216480 kB
Buffers: 2250768 kB
Cached: 12492908 kB
SwapCached: 0 kB
Active: 125394308 kB
Inactive: 2681780 kB
Active(anon): 114261576 kB
Inactive(anon): 456692 kB
Active(file): 11132732 kB
Inactive(file): 2225088 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 8120 kB
Writeback: 20 kB
AnonPages: 113332932 kB
Mapped: 143440 kB
Shmem: 1385764 kB
Slab: 699524 kB
SReclaimable: 562296 kB
SUnreclaim: 137228 kB
KernelStack: 24384 kB
PageTables: 655756 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 65725464 kB
Committed_AS: 6470792 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 40960 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 14716928 kB
DirectMap2M: 116844544 kB
DirectMap1G: 4194304 kB

cat /proc/40339/status
Name: tikv-server
Umask: 0022
State: S (sleeping)
Tgid: 40339
Ngid: 0
Pid: 40339
PPid: 1
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 65536
Groups: 1000
NStgid: 40339
NSpid: 40339
NSpgid: 40339
NSsid: 40339
VmPeak: 185634276 kB
VmSize: 185583056 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 71749592 kB
VmRSS: 59719276 kB
RssAnon: 59714084 kB
RssFile: 5192 kB
RssShmem: 0 kB
VmData: 185305160 kB
VmStk: 264 kB
VmExe: 46204 kB
VmLib: 3028 kB
VmPTE: 327968 kB
VmPMD: 684 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
Threads: 182
SigQ: 0/513382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000180004e43
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: ff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-263
Mems_allowed: 00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 18603
nonvoluntary_ctxt_switches: 98

[Attachment]

Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.

2
| username: wuxiangdong | Original post link

Is it possible that your changes didn’t take effect?

3
| username: wuxiangdong | Original post link

Try using this: set config tikv storage.block-cache.capacity=‘10G’;

4
| username: 饭光小团 | Original post link

Effective. From the monitoring, you can see that block-cache.capacity is less than 10G. In addition, after I executed the command, I could see from the top that the memory usage rate dropped a bit. Previously, block-cache.capacity was 15G.

5
| username: h5n1 | Original post link
6
| username: alfred | Original post link

“The memory-usage-high-water parameter controls the memory limit for TiKV, which defaults to 90% of the system memory. How about adjusting this parameter memory-usage-high-water? It looks like you don’t have much available memory left, MemAvailable: 14216480 kB.”

7
| username: 饭光小团 | Original post link

I have also read this article~ Thanks for sharing. But I just want to know one thing, from the monitoring panel, it doesn’t show that TIKV is using 60G+ of memory, so what exactly is using it?

8
| username: h5n1 | Original post link

This will be available in future versions; the official team has been working on it continuously.

9
| username: 饭光小团 | Original post link

Sorry, I checked the documentation and it seems that there is no such parameter. Are you referring to the memory-limit parameter? TiKV 配置文件描述 | PingCAP 归档文档站

10
| username: 饭光小团 | Original post link

:+1:

11
| username: alfred | Original post link

This is indeed not found in the official documentation, but I saw it in the following article. You can only try to take a look.

12
| username: 饭光小团 | Original post link

I’ll search through the source code to see if this parameter exists.