How many historical versions can MVCC store?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: MVCC可以保存多少历史版本?

| username: 春风十里

  1. Various Timeouts in TiDB | PingCAP Documentation Center

GC Timeout

TiDB’s transaction implementation uses the MVCC (Multi-Version Concurrency Control) mechanism. When new data overwrites old data, the old data is not replaced but retained alongside the new data, distinguished by timestamps. TiDB periodically cleans up unnecessary old data through a GC mechanism.

By default, TiDB ensures that each MVCC version (consistent snapshot) is retained for 10 minutes. Transactions that read data for more than 10 minutes will receive an error GC life time is shorter than transaction duration.

  1. MVCC

Many databases implement Multi-Version Concurrency Control (MVCC), and TiKV is no exception. Imagine a scenario where two clients simultaneously modify the value of a key. Without multi-version control, data locking would be necessary, which could lead to performance and deadlock issues in a distributed environment. TiKV’s MVCC implementation adds version numbers to keys. Simply put, without MVCC, TiKV can be viewed as:

Key1 -> Value
Key2 -> Value
……
KeyN -> Value

With MVCC, TiKV’s key arrangement looks like this:

Key1_Version3 -> Value
Key1_Version2 -> Value
Key1_Version1 -> Value
……
Key2_Version4 -> Value
Key2_Version3 -> Value
Key2_Version2 -> Value
Key2_Version1 -> Value
……
KeyN_Version2 -> Value
KeyN_Version1 -> Value
……

Note that for multiple versions of the same key, the larger version numbers are placed in front, and the smaller ones are placed behind (see the Key-Value section, where keys are arranged in order). This way, when a user retrieves a value using a key + version, they can construct the MVCC key using the key and version, i.e., Key_Version. Then, they can directly use RocksDB’s SeekPrefix(Key_Version) API to locate the first position greater than or equal to this Key_Version.
When users need longer read times, such as when using Mydumper for full backups (Mydumper backs up consistent snapshots), they can adjust the tikv_gc_life_time value in the mysql.tidb table in TiDB to extend the MVCC version retention time. Note that the tikv_gc_life_time configuration affects the entire system immediately. Increasing it will extend the lifespan of all existing snapshots, while decreasing it will shorten the lifespan of all snapshots immediately. Too many MVCC versions can slow down TiKV’s processing efficiency, so after using Mydumper for a full backup, it’s necessary to promptly revert tikv_gc_life_time to its previous setting.

For more information on GC, please refer to the GC Mechanism Overview document.

Transaction Timeout

Garbage collection (GC) does not affect ongoing transactions. However, pessimistic transactions still have limits, including a transaction timeout limit (modifiable via the max-txn-ttl setting under the [performance] category in the TiDB configuration file, defaulting to 60 minutes) and a memory usage limit.

SQL statements like INSERT INTO t10 SELECT * FROM t1 are not affected by GC, but if they exceed the max-txn-ttl time, they will roll back due to timeout.

After reading these two articles, I want to ask if TiDB’s MVCC strictly adheres to a 10-minute (no transaction) retention mechanism, meaning does it have to clean up historical data after 10 minutes? Also, if there are multiple updates within 10 minutes, are they all retained? 10 minutes seems like a short default value.
Databases like Oracle and MySQL have the concept of undo tablespaces. Does TiDB have a similar concept? Or to put it another way, is MVCC in TiDB stored the same way as regular data, or is there a separate TiKV Region or something similar to an undo tablespace?

| username: tidb菜鸟一只 | Original post link

If you set the MVCC retention time to 10 minutes, it strictly retains data only within 10 minutes. However, it doesn’t necessarily clean up immediately after 10 minutes. The GC cleanup also has settings, such as how often it runs and the number of cleanup threads. Multiple updates within 10 minutes will be retained. The default value is indeed relatively short, so you can set it to an appropriate time based on your cluster’s performance.

Oracle and MySQL use undo to handle historical data. The principle of undo is to store the modified data separately in undo, and if needed, retrieve it from there. However, TiDB uses MVCC, which is a different concept. TiDB’s MVCC essentially timestamps each piece of stored data and stores it in the region of your current table. When you perform a normal query, it directly returns the data with the latest timestamp. If you query historical data, it finds and returns the data with the corresponding timestamp, without storing it separately in other regions.

| username: 春风十里 | Original post link

Okay, thank you for the reply. I see that your response mentions that MVCC defaults to 10 minutes, with all updates retained, and there is no concept of undo tablespace, which means it is a local multi-version.

| username: zhanggame1 | Original post link

The underlying layer of TiDB is key-value (KV) without the concept of tablespace. When data is modified, it actually inserts a new modified KV. The GC operation has a scheduled time, which is 10 minutes by default. The expiration of KV also has a time difference, which is also 10 minutes by default. Both can be changed.

| username: Anna | Original post link

Learned.

| username: Anna | Original post link

Please refer to this

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.