Can the Follower node of the PD component provide timestamps?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: PD 组件的Follower节点可以提供时间戳嘛

| username: TiDBer_2i1SqvUB

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path]
[Encountered Problem: Problem Phenomenon and Impact]
I want to deploy TiDB across multiple centers. In a multi-data center scenario, the PD leader is located in one data center, and the latency between data centers will cause an increase in TSO request latency. Is it possible for PD Follower nodes to provide timestamps and ensure they are ordered with the timestamps provided by the PD Leader? This way, the PD Follower nodes can directly allocate timestamps to the TiDB components in their respective centers without needing to obtain timestamps across centers.


To ensure performance, PD does not generate a TSO for each request but pre-allocates a distributable time window. The time window is the current time and the TSO 3 seconds after the current time, stored in etcd. Subsequently, TSO can be allocated from the window. We periodically pre-allocate the time within the window to the PD Follower nodes, so the timestamps on all PD Follower nodes are uniformly generated and ordered by the Leader. Each data center can then directly obtain timestamp information from its local PD Follower node. I wonder if this approach is feasible. I hope the experts can provide some guidance.

| username: h5n1 | Original post link

Currently, it is not possible, but you can configure follower forwarding.

tidb_enable_tso_follower_proxy introduced from version v5.3.0

  • Scope: GLOBAL
  • Persisted to the cluster: Yes
  • Type: Boolean
  • Default value: OFF
  • This variable is used to enable the TSO Follower Proxy feature. When this value is OFF, TiDB will only obtain TSO from the PD leader. After enabling this feature, TiDB will evenly distribute TSO requests to all PD nodes, forwarding TSO requests through PD followers, thereby reducing the CPU pressure on the PD leader.
  • Scenarios suitable for enabling TSO Follower Proxy:
    • The PD leader reaches a CPU bottleneck due to high-pressure TSO requests, resulting in high latency for TSO RPC requests.
    • There are many TiDB instances in the cluster, and increasing the tidb_tso_client_batch_max_wait_time does not alleviate the high latency of TSO RPC requests.
| username: TiDBer_2i1SqvUB | Original post link

Sure, boss; I just wanted to ask if there’s an issue with this approach if I modify the code :joy:

| username: h5n1 | Original post link

If you can modify the code, you are the expert. You can take a look at this first:

| username: zhanggame1 | Original post link

TSO requires strict time increment, which can only be handled centrally. Theoretically, it is impossible to separate it, so don’t overthink it.

| username: MrSylar | Original post link

I think the question is about local TSO, right? TiDB 6.0: Making TSO More Efficient | TiDB Books

| username: MrSylar | Original post link

I suddenly found that there is no local TSO in the official documentation.

| username: TiDBer_2i1SqvUB | Original post link

Okay, I would like to ask, these timestamps are first generated by the PD Leader node, so they are ordered, and then these ordered timestamps are distributed to different Followers. These timestamps are also ordered in a distributed scenario, right? What is the difference compared to getting timestamps directly from the PD Leader? Is it that the timestamp I get later is definitely greater than the one I got earlier? And the method I mentioned of pulling timestamps from Followers cannot achieve global order, right? It cannot achieve external consistency.

Just like this, the data center on the left sets x to 100 at timestamp 1290, and then the data center on the right reads x at timestamp 1280 and finds that it cannot read it. But from an external perspective, it should be readable because the operation on the left was done first. This method does not conform to external consistency, right?

| username: TiDBer_2i1SqvUB | Original post link

Yep, I saw it before. By using LocalTSO, the occurrence of global transactions is reduced.

| username: TiDBer_2i1SqvUB | Original post link

TiDB BOOKS are great!

| username: zhanggame1 | Original post link

The Follower function only proxies TSO requests and does not cache TSO itself.

| username: zhanggame1 | Original post link

As for localTSO, it is just an experimental feature. The enable-local-tso parameter is not even mentioned in the official PD parameter documentation.

| username: redgame | Original post link

Does not directly provide timestamp functionality.

| username: zhanggame1 | Original post link

The document you posted mentions at the end, “local TSO as an experimental feature still needs improvement, and in the TPCC test, enabling this feature results in a large number of primary key duplication errors.”

This thing is completely unusable.

| username: MrSylar | Original post link

Yes, I have also studied it. I didn’t have an in-depth understanding before. It was quite impressive when I learned about this feature.

| username: cassblanca | Original post link

A key point of knowledge