After Adding a TiKV Node, Cluster Latency Increases - Seeking Solutions

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 增加一台TiKV节点后，集群延迟变高，请教如何解决

| username: TiDBer_O0lRJ4Ee

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] v4.0.7
[Encountered Problem: Problem Phenomenon and Impact]
Added a TiKV node to the cluster, data is being balanced, and latency is very high. The following adjustments were made but had no effect:
Slow down
set config pd schedule.leader-schedule-limit=1;
set config pd schedule.merge-schedule-limit= 2;
set config pd schedule.max-pending-peer-count= 1;
set config pd schedule.replica-schedule-limit= 1;
set config pd schedule.max-snapshot-count = 1;

[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

image
image1380×640 93.1 KB

[Attachments: Screenshots/Logs/Monitoring]

image
image1380×560 140 KB

2
| username: tidb菜鸟一只 | Original post link

Which nodes have 100% memory usage?

3
| username: 我是咖啡哥 | Original post link

Check the slow SQL and see if the business SQL has significantly slowed down. If the business SQL is not affected, then it doesn’t matter. That delay includes some background SQL.

4
| username: TiDBer_jYQINSnf | Original post link

Under the store limit restriction.
Additionally, region migration can cause the TiDB region cache to become invalid.
Region migration can also reduce the block cache hit rate of RocksDB.
Region migration also consumes disk I/O.

Check the frequency of operator generation from the PD dashboard.

5
| username: redgame | Original post link

You can use the balance-region command to manually trigger the balancing of Regions.

6
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.