TiFlash Table Synchronization is Slow

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash 表同步慢

| username: Liantao001

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path]
[Encountered Problem: TIDB Tiflash table synchronization is slow
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page



tiflash_error.log screenshot


tiflash_error_30.txt (3.3 MB)

Now the disk usage has decreased, but synchronization is still slow, and tiflash_error.log has the following error:
[FLASH:Coprocessor:RegionError] Region 3931443898 is unavailable at 9: (while doing learner read for table, logical table_id: 10793275)

| username: 这里介绍不了我 | Original post link

Are these tables very large?

| username: zhh_912 | Original post link

Possible reasons for slow TiFlash table synchronization include large data volume, network latency, insufficient TiFlash node performance, and connection bottlenecks between TiDB and TiFlash. It is necessary to check and optimize data distribution, network connections, hardware performance, and adjust configuration parameters to improve synchronization speed.

| username: TiDBer_小阿飞 | Original post link

What is the hardware configuration of the TiFlash node? Is the memory sufficient?

| username: WinterLiu | Original post link

The IO performance is poor, right?

| username: 小龙虾爱大龙虾 | Original post link

Check the resources and adjust the scheduling-related parameters. Refer to: 构建 TiFlash 副本 | PingCAP 文档中心

| username: Lloyd-Pottiger | Original post link

The disk is too full, so PD stopped scheduling.

  1. Replace with a larger disk
  2. Set storage.main.capacity to be smaller than the disk space
| username: Liantao001 | Original post link

The disk usage of the node has decreased now, but the synchronization is still very slow.

| username: zhaokede | Original post link

Replace the SSD solid-state drive.

| username: Billmay表妹 | Original post link

  1. The configuration is a bit low.
  2. It might be better to switch the file system to ext4.
| username: Liantao001 | Original post link

Does this error have any impact?

| username: WalterWj | Original post link

Your space usage is too high. By default, scheduling stops at 70%, if I remember correctly. You can check the pd-ctl ratio configuration.

If the scheduling is too slow, you can increase the store limit for TiFlash. The default is 30, you can change it to 200.

You should first confirm whether it has stopped scheduling or if the scheduling is just slow.