TiKV nodes occasionally experience high iowait, causing cluster lag

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv节点久不久就出现iowait高,导致集群卡顿

| username: TiDBer_vC2lhR9G

【TiDB Usage Environment】Production Environment
【TiDB Version】6.5.1
Experts, I need some advice. The TiKV nodes occasionally experience high iowait, causing the cluster to lag. What could be the issue?

| username: tidb狂热爱好者 | Original post link

Method 1: Increase IO
Choose local disks on Alibaba Cloud, the read speed can reach around 1000m. Do not choose cloud SSD.
Method 2: Reduce reads
Go to the slow SQL top panel on the dashboard and optimize the top slow SQL.

| username: 有猫万事足 | Original post link

“Appears from time to time”

| username: TiDBer_vC2lhR9G | Original post link

It seems like my backup has failed. Could it be related to this?
checkpoint[global]: 2023-10-05 10:03:04.679 +0800; gap=1656h53m12s

| username: TiDBer_vC2lhR9G | Original post link

It is possible that every one or two hours, the iowait becomes very high.

| username: oceanzhang | Original post link

  1. Identify the statements with high I/O or use higher performance disks.
| username: oceanzhang | Original post link

Just run for a short period of time.

| username: 有猫万事足 | Original post link

Okay, got it. Is it this particular TiKV or all TiKVs?

If it’s just this one, you might want to check the TopSQL interface to see what this TiKV is currently executing.

| username: andone | Original post link

See if there is a pattern. If there is a pattern, consider whether there are scheduled batch jobs or timed tasks.

| username: 小龙虾爱大龙虾 | Original post link

Analyze the system and optimize it.

| username: TiDBer_vC2lhR9G | Original post link

Resolved, it was the log backup that failed. The task kept maintaining the checkpoint, probably couldn’t hold on anymore.

| username: h5n1 | Original post link

What is log backup?

| username: dba远航 | Original post link

RAFT logs?

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.