TiDB Node Expansion Results in Slow Query Performance

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 扩充节点 出现查询比较慢

| username: jerry

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 5.4
[Reproduction Path] TiDB horizontal scaling, adding TiFlash and TiKV
[Encountered Problem] After scaling, queries become slower, and backend query interfaces experience timeouts. It is suspected that TiKV is performing load balancing. How should this issue be investigated?
[Resource Configuration] Added node memory 250 GiB, CPU 40 cores
[Attachments: Screenshots/Logs/Monitoring]

2
| username: 我是咖啡哥 | Original post link

Scaling out and load balancing can indeed have some impact. Generally, if it’s not urgent, it’s best to do it during off-peak hours. You can check the monitoring of CPU, memory, and IO.

3
| username: jerry | Original post link

[quote=“我是咖啡哥, post:2, topic:1005505”]
看看CPU、内存、IO
[/quote] CPU, memory, and IO are all very low. The issue has been present for more than a day.

4
| username: 裤衩儿飞上天 | Original post link
  1. After adding the TiFlash node, did you build the TiFlash replica?
  2. Regarding TiKV load balancing, you can check the monitoring to see if the regions are already balanced.
5
| username: jerry | Original post link
  1. Deleted the previous replica and reconstructed a new one.
  2. Not quite sure about the second issue. If it’s due to imbalance, is there any quick solution? The current queries are very slow.
6
| username: tidb菜鸟一只 | Original post link

Adding a TiFlash node probably hasn’t added a TiFlash replica yet, so its impact is minimal. Mainly, just check if the TiKV balancing has finished.

7
| username: jerry | Original post link

How to check if TiKV balancing has finished? Please advise.

8
| username: 裤衩儿飞上天 | Original post link
  1. Has the new TiFlash replica been built?
  2. You can refer to PD Control User Guide | PingCAP Documentation Center. Remember to revert to the original value after balancing is completed.