What Causes 99% High Latency in Monitoring?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 监控中99%延迟高是什么原因

| username: TiDBer_Y2d2kiJh

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] v5.4.0 2tidb 3pd 3tikv 2ha
[Reproduction Path] Occasionally, the 99% latency is high in the monitoring, previously it was the 99.9% latency that was high. How can I locate this issue, as shown in the figure.
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]

| username: 像风一样的男子 | Original post link

This situation is most likely caused by slow queries.

| username: songxuecheng | Original post link

Caused by the drop in QPS

| username: TiDBer_Y2d2kiJh | Original post link

Isn’t it that higher latency leads to lower QPS?

| username: Miracle | Original post link

You can check the slow queries for the corresponding time period.

| username: Fly-bird | Original post link

This latency isn’t considered high, right? Check the business or slow queries during that time period.

| username: songxuecheng | Original post link

If there is no noticeable slowdown in the business, it is likely because the QPS of the faster-running SQL queries is lower than before, which causes the 99th percentile to increase.

| username: TiDBer_Y2d2kiJh | Original post link

An update ran for 50 seconds.

| username: TiDBer_Y2d2kiJh | Original post link

At the same time that the latency is high, the service response time has also increased.

| username: songxuecheng | Original post link

You need to check for expensive SQL and whether there are any locks.

| username: 像风一样的男子 | Original post link

Is the table being updated a highly concurrent table? Locking the table leads to a decrease in QPS.

| username: TiDBer_Y2d2kiJh | Original post link

At that time, the query showed no lock waits or deadlocks.

| username: 像风一样的男子 | Original post link

Your latency isn’t very high and shouldn’t cause a drop in QPS. Check the topsql during this period to see what’s happening.