TiKV Latency Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Tikv延时问题

| username: vincent

[Test Environment for TiKV]
[TiKV Version] v5.4.0
[Usage] We are using raw TiKV for testing, and the transaction processing is completed with the help of the Golang version of the TiKV client (GitHub - tikv/client-go: Go client for TiKV).

In testing the impact of the number of GRPC connections between the TiKV client and TiKV server on performance, we found that when setting GrpcConnectionCount higher, the P99 tail latency actually increased.

[Test Scenario] The transaction types that expose latency in our tests are named getObjectInfo, deleteObjectInfo, and putObjectInfo. We specified a 12K QPS load test, and the load test client is only on one machine.
We found:
When GrpcConnectionCount=1, after stabilizing, the P99 tail latency is the smallest.
When GrpcConnectionCount=16, after stabilizing, the P99 tail latency is the largest.
In the load test scenario, the machine’s resources were not heavily used, meaning the hardware resources of the machine were relatively abundant.

What is the best practice for setting the number of GRPC connections?

| username: redgame | Original post link

This value depends on concurrency and latency. Regularly monitor and evaluate the performance of the number of connections, and adjust and optimize as needed.

| username: 有猫万事足 | Original post link

I think you can refer to the gRPC thread pool section in this document: TiKV 线程池性能调优 | PingCAP 文档中心.

My guess is that you increased the number of client-side connections, but the thread pool on the TiKV server side remains the same. So the client connections end up with additional waiting time.

Especially since the document mentions that the default number of server-side threads is 5, and in your graph, there is a noticeable increase in latency when the number exceeds 4, which corresponds to this. You can try adjusting it to see how it affects performance.

| username: vincent | Original post link

I’ll try this again.

| username: vincent | Original post link

Is there a specific guide document you can share?

| username: cy6301567 | Original post link

This is well explained.