Do different TiDB Servers need to load from storage when accessing the same data?

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 不同TiDB Server 上访问相同数据,是否需要到存储加载?

| username: 春风十里

[TiDB Usage Environment] Testing
[TiDB Version] V7.1.2

When accessing the same data on different TiDB Servers, does it need to be loaded from storage?
For example, in a cluster with three TiDB Servers, if a program connects to TiDB Server 1 to access table T1 and loads it into the memory of TiDB Server 1, and at the same time another program connects to TiDB Server 2 and also needs to access table T1. Assuming that T1 is entirely in the memory of TiDB Server 1 at this time, when TiDB Server 2 also needs to access table T1, will the data be transferred from TiDB Server 1 to TiDB Server 2, or will it be fetched from TiKV?

| username: zhanggame1 | Original post link

Retrieve from TiKV. TiDB has limited memory and cannot store data from too many tables, so operator pushdown is necessary. Some WHERE filtering, aggregate functions, etc., are completed on TiKV, and then summarized and processed in TiDB.

| username: 小龙虾爱大龙虾 | Original post link

Yes, it will fetch from TiKV.

| username: 普罗米修斯 | Original post link

Retrieve from TiKV

| username: buptzhoutian | Original post link

Version v6 introduced cached tables.

The entire table’s data is loaded into the TiDB server’s memory, allowing table data to be retrieved directly from memory and avoiding fetching data from TiKV.

| username: TiDB_C罗 | Original post link

Is the cache table cached on each TiDB?

| username: buptzhoutian | Original post link

It probably isn’t. Why don’t you trace it and test it :smile:

| username: Kongdom | Original post link

The second connection must be obtained from TiKV because the TSO of the second and the first are different. Different TSOs mean reading different versions of the data. :yum:

| username: 春风十里 | Original post link

Theoretically, the speed of memory plus network should be faster than that of disk plus network. If the data on the first TiDB Server is the latest, I feel that the performance will be better when it is transmitted from this machine.

| username: zhanggame1 | Original post link

It’s a bit difficult to determine if a distributed database is new.

| username: zhanggame1 | Original post link

This cache table is really not very useful, difficult to modify, and not much faster.

| username: 有猫万事足 | Original post link

You’re right, so determining whether the data is up-to-date is indeed a problem.

The Raft protocol mentions two approaches, both implemented in TiKV, not in TiDB.
TiDB instances do not communicate with each other and are stateless services.

The caching design of the entire TiDB system is also multi-layered. TiDB has a cache, and so does TiKV. Therefore, fetching data directly from TiKV, following the Raft process, is actually about the same efficiency as fetching it from another TiDB instance.

| username: dba远航 | Original post link

TiDB servers are unaware of each other, so they do not communicate with each other.

| username: Kongdom | Original post link

Only the data in TiKV is the latest, and it is impossible to determine if the data in TiDB is the latest.

| username: TiDBer_小阿飞 | Original post link

It is definitely TiKV. Even for cached tables, it is a single TiDB server node responsible for translating SQL into Key-Value operations, forwarding them to the shared distributed Key-Value storage layer TiKV, assembling the results returned by TiKV, and finally returning the query results to the client. The nodes are completely equal to each other.

| username: forever | Original post link

There will still be contention when fetching from another tidb-server, similar to how the biggest issue with RAC is GC contention.

| username: 春风十里 | Original post link

You mentioned Oracle RAC’s gc contention, but this is just a minor issue of RAC’s content fusion technology, Cache Fusion. Cache Fusion extends the information interaction method based on the shared-disk architecture, allowing different nodes to share the internal buffer of the database through the interconnect network. Data is directly transferred from one node’s buffer to other nodes, avoiding read and write to the shared disk.

Its underlying design logic is that the speed of disk + fiber switch is slower than the speed of memory + network. Fundamentally, it is to reduce disk IO and improve performance. Practice has proven that it is relatively stable in multi-active architectures. gc contention is just a performance issue caused by implementing multi-active, but it does not negate its value.

Of course, ORACLE RAC is a shared-disk architecture, while TiDB is a shared-nothing architecture. The technical routes are different and cannot be directly compared.

What I want to ask is, when TiDB executes SQL, it first accesses the PD to obtain region information, which means it naturally knows the distribution of regions, which is the leader, and which is the follower. In this way, should it also know which region is already on which TiDB server? If so, can TiDB also use “memory fusion technology” to reduce disk IO and improve performance?

| username: 有猫万事足 | Original post link

Your guess is roughly correct.

The implementation you are asking about should be in client-go.

TiDB uses this as the client to access TiKV.
There is a region information cache (region cache) in client-go, so the client knows which TiKV a certain region is on. Of course, there are times when it gets scheduled away. If it gets scheduled away and an error “region xxx is not leader” occurs, the region cache for the corresponding ID will be invalidated, and the client will re-query the region information from PD.

| username: forever | Original post link

The region information on PD is reported by the nodes. To know which regions are cached by the TiDB server, the TiDB server needs to report the cache information to PD. However, the TiKV nodes themselves also have caches, and the latest data retrieved from the TiKV nodes is also obtained from memory. Therefore, the “memory fusion technology” between TiDBs doesn’t seem to have much significance.

| username: zhanggame1 | Original post link

TiDB data caching mainly relies on TiKV for caching. In theory, TiDB knows which TiKV holds the data. Different TiKVs have different data, and by default, it only queries the leader of that region.