I deployed a monolithic simulation cluster and found that the memory usage increases rapidly when querying data continuously. I understand that the LSM-tree data interface inherently has space amplification. However, I noticed that only the C0 layer of the LSM-tree is in memory. When the data reaches a certain amount, the C0 layer merges into the lower Cn layers, which should primarily occupy disk space. Given this merging operation, why does the memory usage also keep increasing?
Are you referring to TiDB server or TiKV memory usage? It’s best to check with top and sort by memory using shift+m to see the SHR column and identify which process is using the memory. In my previous tests, TiKV memory usage was generally stable, while TiDB server memory usage often increased rapidly.
How do you determine that the memory is occupied by the LSM tree? Look at the memory usage of various components clearly. Distributed systems definitely occupy more memory than single-node systems.
The default value of tidb_enable_clustered_index is INT_ONLY, which means that the clustered index is enabled only for tables with integer primary keys. If you want to enable the clustered index for all tables, you need to set this parameter to ON.
Single-node deployment: 1 TiDB + 3 TiKV + 1 iFlash + 1 MySQL + N other components.
It’s already good enough if it can run normally. For this kind of deployment, it’s recommended to only look at functional features. If you want to evaluate performance, this deployment architecture is the biggest issue, as the resource competition among all components is too intense. To evaluate performance, deploy them on separate nodes.
The resource usage for a single-machine deployment is at a normal level. The tidb-server is written in Go, and memory release involves garbage collection (GC), which takes some time.
Sure, thank you. I’m just not clear about one thing. When inserting data, under the same conditions, TiDB obviously uses more memory than MySQL. I wonder if this memory usage is due to the space amplification caused by the LSM-tree?
The high memory usage of TiKV is usually caused by the block-cache parameter being too large.
You can set this value smaller, especially in scenarios where multiple TiKVs are deployed together. This parameter is particularly important and needs to be adjusted to an appropriate memory value.
Specifically, you can search the forum for how to optimize this parameter.
|
username: TiDBer_JlY1JCJ5
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
For a single machine, it is recommended to use either playground or a single TiKV.