Frequent gRPC error in TiKV-rust client: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiKV-rust client频繁出现gRPC error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded

| username: TiKV新用户

[TiDB Usage Environment] Test environment
[TiDB Version] v5.4.2
[Encountered Problem] Using tikv-rust-client to access the TiKV cluster for reading and writing data, using the transaction interface, occasionally experiencing timeouts for a period of time.
[Reproduction Path] What operations were performed to encounter the problem
Just reading and writing to TiKV
[Problem Phenomenon and Impact]
Occasionally, requests sent to TiKV report gRPC error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded

[Attachment]

Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.

tikv v5.4.2
The TiKV cluster has 3 TiKV nodes and 3 PD nodes, no TiDB nodes, no monitoring.
The machine configuration is 8c16g, and TiKV is configured with resource_control.memory_limit set to 8g.

What could be the possible reasons for this timeout? Is it an issue with the rust-client or the cluster? In the absence of TiDB, does the client need to actively call the GC interface?

2
| username: 近墨者zyl | Original post link

Waiting for the expert’s reply, learning.

3
| username: xfworld | Original post link

Transactions have a default maximum time limit. If the operation is not completed within this time frame, it will be handled by the client as a deadline.

Suggestions for adjustment:

  • Reduce the read/write range to decrease transaction processing time.
  • Adjust the maximum duration of the transaction, which may lead to other issues (such as memory overflow).
4
| username: TiKV新用户 | Original post link

Thank you for the reply.
I adjusted the grpc timeout to 3 seconds. Is it possible that a request cannot be processed within 3 seconds?
Additionally, may I ask, in the absence of a TiDB instance, will PD proactively call TiKV’s GC? Or do I need to actively call GC through the TiKV client?

5
| username: 近墨者zyl | Original post link

GC is initiated by TiDB and executed by TiKV, and it has nothing to do with PD. PD has no relation to the MVCC mechanism.

6
| username: 近墨者zyl | Original post link

Study case, :+1:

7
| username: xfworld | Original post link

Whether the request can be processed depends on the method and scale of the data processing, which you need to control yourself.

Without a TiDB node instance, GC will not be initiated by default. The simplest way is to add such a node, and that’s it. :star_struck:

8
| username: TiKV新用户 | Original post link

Understood, thank you. Indeed, adding a TiDB node is the simplest solution.

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.