A Question About Memory Usage Calculation in RocksDB

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: RocksDB 的内存占用计算中1个疑问

| username: 白丁2023

[TiDB Usage Environment] Production Environment / Testing / PoC
Testing
[TiDB Version]
v6.1.0
[Reproduction Path] What operations were performed when the issue occurred
A question about memory usage calculation in RocksDB
[Encountered Issue: Problem Phenomenon and Impact]
Document description:
Data written to RocksDB is written to the MemTable. When the size of a MemTable exceeds 128MB, it switches to a new MemTable for writing. There are a total of 2 RocksDB instances in TiKV, with a total of 4 ColumnFamilies. The size limit for a single MemTable in each ColumnFamily is 128MB, and a maximum of 5 MemTables are allowed, otherwise, it will block foreground writes. Therefore, the maximum memory occupied by this part is 4 x 5 x 128MB = 2.5GB.

Question 1:
There are a total of 2 RocksDB instances in TiKV, with a total of 4 ColumnFamilies. Is it 4 ColumnFamilies or 5 ColumnFamilies?

[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]

Image
Image880×397 11 KB

Document: RocksDB 简介 | PingCAP 文档中心

2
| username: h5n1 | Original post link

The official documentation is probably wrong by 5.

3
| username: 白丁2023 | Original post link

Could it be that before version 5.4, when raftdb was not enabled, there were 4 CFs?

4
| username: magic | Original post link

The documentation says: One rocksdb has 1, another has 4 cf, a total of 5.

5
| username: 白丁2023 | Original post link

Can you send the link?

6
| username: h5n1 | Original post link

Always there.

7
| username: magic | Original post link

Regarding the link you posted

8
| username: TiDBer_jYQINSnf | Original post link

There are 4 column families (cf) in rocksdb-kv, 3 of which are used to store actual data: write, default, and lock. The other one is raft, which is used to store the local state of the region, such as the region apply index and the state of the region.
In rocksdb-raft, there is one default cf that stores raft logs.

A total of 5.
The official documentation should not have introduced the raft cf in rocksdb-kv.