Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: TiCDC 同步到下游TiDB集群时Sorter模块输出速率过低
[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.0
Help, help!!! We are experiencing delays in TiCDC synchronization to the downstream cluster. Upon observation, we found that the synchronization rate can only reach a few hundred to a thousand per second, far from the official claim of 50K/s. Recently, we discovered that the output rate of the Sorter module in CDC is very low. Please help us identify the bottleneck in this sorter. As shown in Figure 1, the puller can fetch over 10,000 records per second, but in Figure 2, the Sorter module outputs only 1,000.
Figure 1
Figure 2
I see that the officially recommended sorting engine is DB sorter, but no matter how we set it, it always defaults to Unified. I’m not sure if this is the reason for the slow sorter rate.
Is the write load of the business mainly concentrated on a few tables?
Unified Sorter will store temporary files on the disk, and if it’s a mechanical disk, the performance will be very poor.
Yes, there are only a few tables being synchronized, and each has a data volume of over 200 million.
I think the problem is that the TiDB version you are using is too old. You can try upgrading to the latest version and see if the issue persists.
So how can I switch to DB Sorter?
If not actively configured, the default is DB sorter.
My production environment cluster: Upgraded from version v5.4.0 to v6.5.0
- Use the following command to create a new synchronization task without specifying the sort engine during creation:
tiup cdc cli changefeed create --server=http://10.3.9.16:8300 --sink-uri="mysql://root:xxxxx@10.3.9.3:4000/?worker-count=64&max-txn-row=1024" --changefeed-id="bj-sjzt-to-slave-04" --config="./bj_sjzt_to_slave_04.toml" --start-ts='450536745467904001'
- After creation, querying this synchronization task shows that the sort engine is still using: unified. How can this be resolved?
Starting component `cdc`: /root/.tiup/components/cdc/v6.5.0/cdc cli changefeed query --pd=http://10.3.8.230:2379 --changefeed-id bj-sjzt-to-slave-04
{
"upstream_id": 7165868112300891779,
"namespace": "default",
"id": "bj-sjzt-to-slave-04",
"sink_uri": "mysql://root:xxxxx@10.3.9.3:4000/?worker-count=64\u0026max-txn-row=1024",
...
"create_time": "2024-06-18 11:53:02.997",
"start_ts": 450536745467904001,
"resolved_ts": 450563299399696391,
"target_ts": 0,
"checkpoint_tso": 450563299333898296,
"checkpoint_time": "2024-06-19 10:03:05.960",
"sort_engine": "unified",
"state": "normal",
"error": null,
"error_history": null,
"creator_version": "",
"task_status": [
{
"capture_id": "dc1f5121-d423-4356-b1c2-89f56eb795f1",
"table_ids": [],
"table_operations": null
},
{
"capture_id": "864ce151-efd8-4bd7-950b-adf897eb386f",
"table_ids": [
14257
],
"table_operations": null
}
]
}
Upgraded from a lower version, using the previous default configuration. Try explicitly changing the parameters under cdc in edit-config and then reloading. I haven’t tried it. Not sure if it will work.
How do I modify it in edit-config? I tried several times before, but it didn’t work.
According to the query, the returned sort-engine is unified. Does this indicate DB Sorter or UnifiedSorter?
|
username: kevinY
|
username: 
|
username: 
|
username: 
|
username: