Abnormal Duration in TiKV

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv的duration异常

| username: dbaspace

[TiDB Usage Environment] Production Environment
After an abnormal reboot of the host, the tikv duration response was very slow upon startup, affecting the tidb duration. As shown in the figure:
tikv duration:


tidb duration: Red arrow, response is very slow


The read and write operations of the cluster have slowed down. Currently, the cluster is stable by reducing the scheduling, but the tikv duration response is still very slow.

Is there any way to keep the tikv duration at a relatively low value?

| username: tidb菜鸟一只 | Original post link

It’s not working, let’s remove the TiKV on the machine that was restarted first. The KV request duration has reached 1 minute…

| username: dbaspace | Original post link

Previously, I didn’t notice that a host reboot would cause this issue. It has happened twice recently, and evicting TiKV resolves it. The logs show elections, but I’m not sure if there are any other parameters that can control this?

| username: dbaspace | Original post link

This node’s leader has recovered by one-third.

| username: ti-tiger | Original post link

I think if possible, you should upgrade the version first since v3.0.19 is indeed a bit outdated. :grin:

| username: dbaspace | Original post link

At least it works fine now, but upgrading will make it even worse.

| username: 像风一样的男子 | Original post link

Does 3.0 not have tiup?

| username: dbaspace | Original post link

No. ansible :grin:

| username: ajin0514 | Original post link

Recommend a higher version.

| username: h5n1 | Original post link

What is the current balance of leaders and regions? Check the network latency monitoring with black exporter. Look at the tikv-detail monitoring to see which type of backoff is higher.

| username: ajin0514 | Original post link

Take a look at the load balancer.

| username: 路在何chu | Original post link

However, not upgrading will eventually lead to failure. Upgrade sooner rather than later.

| username: ajin0514 | Original post link

It is recommended to upgrade the version.

| username: dbaspace | Original post link

After upgrading the kernel version, bringing TiKV back online is normal.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.