What could be the reasons for significant latency fluctuations when QPS does not change significantly? How to investigate the causes? Thank you

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 请问在QPS没有明显变化的情况下,延迟大幅度波动可能有哪些原因?怎么调查原因?谢谢。

| username: TiDBer_56p09FBa

【TiDB Usage Environment】Production Environment
【TiDB Version】V6.1.0
【Reproduction Path】Occurs with select/update/insert
【Encountered Problem: Phenomenon and Impact】As mentioned, intermittent latency increase
【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】

| username: TiDBer_56p09FBa | Original post link

The image is not displayed. Please provide the text you need translated.

| username: xfworld | Original post link

Check the traffic visualization to see if there are any particularly bright spots.

| username: h5n1 | Original post link

The significant change is in the p999 latency, which is usually caused by individual slow SQL queries. Slow prewrite is largely related to disk I/O and network issues. You can refer to the troubleshooting process for slow writes at TiDB 写入慢流程排查系列(一)— 前言 - TiDB 的问答社区.

| username: TiDBer_56p09FBa | Original post link

At the time of the failure, there were many “key is locked” messages.

| username: TiDBer_56p09FBa | Original post link

The image is not available for translation. Please provide the text content directly.

| username: TiDBer_56p09FBa | Original post link

What does p99.9% mean? What does it represent? Thank you.

| username: h5n1 | Original post link

99.9% of the latency does not exceed the curve value, which can be compared with p99.

| username: TiDBer_56p09FBa | Original post link

The image you provided is not accessible. Please provide the text you need translated.

| username: TiDBer_56p09FBa | Original post link

There are several p999 spikes every day causing business alerts. Can you help provide some troubleshooting directions?

| username: TiDBer_56p09FBa | Original post link

What does the following content in the TIDB log mean? What does it represent?
wait response is cancelled

| username: ealam_小羽 | Original post link

Check the slow queries during the spike period in TiDB Dashboard?

| username: xfworld | Original post link

Click on those highlighted bars to see which tables they are…