Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: tidb.log 中有大量的 switch region leader to specific 正常吗？
|
username: xiaohetao
As shown in the above tidb.log, there are a large number of “switch region leader to specific”. Is this normal?
Are you importing data? According to similar posts, it doesn’t seem to have much impact.
This log indicates that TiDB sent a region request to TiKV, but the accessed region is not the leader. This might be caused by an expired region cache. It will automatically retry, and if it doesn’t happen frequently, it should not impact the business.
I see that each instance has as many as 8000 and as few as a few hundred. Is 8000 considered a lot or a little? What criteria are used for evaluation?
Check the Grafana monitoring to see if a TiKV node went down, causing all the leaders to move away and then rebalance back.
There was a scaling operation, but that was done in November last year.
Do you mean that TiKV crashed and then recovered by itself?
It’s possible, for example, due to network jitter or excessive pressure on TiKV.
From the monitoring, there are no abnormal phenomena in the cluster, and there are no signs of KV crashing.
That wasn’t caused by scaling in or out; it should be some other reason leading to a large number of leader migrations. Did it resolve itself after a period of time, or is it still reporting this issue continuously?
Looking at PD, there is an operator page that monitors the generation, checking, processing, and cancellation of operators. Check what the majority of the reasons for processing are.
The system has very little business volume, so it shouldn’t be due to excessive pressure on TiKV.
I’ve compared the last few days, and they are roughly the same.
This is mainly about writing hotspots and balancing regions. If this affects TiDB, you can limit the speed with the following command:
|
username: 
|
username: 
|
username: 
|
username: pd-ctl -i
store limit all 5
The default store limit is 15, which means each store handles 15 region actions per minute. Limiting it to 5 can reduce some load. If there are no anomalies, no action is needed. The reason is to adjust the regions with hotspot writes.
Will this cause application connection interruptions? Or will it cause excessive application response times?
The response time is long.
TiDB itself has a cache for the region leader’s location. If the region’s location changes, the TiDB node needs to retrieve it again from PD, which increases the latency.
Other than that, there’s no significant impact.
Are you looking at the transfer-hot-write-leader metric in the picture?