Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: TiKV cdc组件 panic The CommitTs must be greater than the resolvedTs
|
username: Timber
[TiKV Version]
v6.1.0
[Problem Encountered]
Pulling historical data through the tidbkv component and inserting data into TiKV. Using the CDC component puller for data synchronization, a panic occurs on the CDC client side with the message “The CommitTs must be greater than the resolvedTs.”
I found a similar bug that has been fixed in version 5.2, but I did not find information about it being merged into version 6.1. Could you please clarify if this issue is the same as the one mentioned here: kv client receives commit event less then resolved ts · Issue #3075 · pingcap/tiflow · GitHub? Thank you very much for your attention and response.
[Reproduction Path]
The CDC client restores a large amount of historical data through the Iter interface while inserting data into TiKV. The CDC puller component simultaneously pulls data changes.
[Phenomenon and Impact]
Phenomenon: The CDC puller client panics, causing the service to restart. The client first logs several “resolved ts fallen back” messages, followed by a panic. Detailed logs are as follows:
[2022/09/06 05:20:24.072 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namespace=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]
[2022/09/06 05:20:24.110 +08:00] [WARN] [region_worker.go:360] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [namespace=xxx] [changefeed=] [addr=A16969629704832:17513] [regionID=76069] [span=“[0000000000000001ff00000d1000000182ffffbbfc023765007cff846a000000000000f9, 0000000000000001ff00000d2000000182ffff6a8dfe3765ff86ff9f43000000000000f9)”] [duration=6h20m8.3s] [lastEvent=37.565138ms] [resolvedTs=435785568274874381]
[2022/09/06 05:20:29.078 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namespace=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]
[2022/09/06 05:20:30.079 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namespace=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]
[2022/09/06 05:20:31.080 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namespace=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]
[2022/09/06 05:20:31.787 +08:00] [PANIC] [region_worker.go:716] [“The CommitTs must be greater than the resolvedTs”] [EventType=COMMIT] [CommitTs=435760862156032795] [resolvedTs=435785568274874381] [regionID=76069] [stack=“github.com/pingcap/tiflow/cdc/kv.(*regionWorker).handleEventEntry
\t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:716
github.com/pingcap/tiflow/cdc/kv.(*regionWorker).processEvent
\t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:398
github.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler
\t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:495
github.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4
\t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:633
The Go Programming Language
\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75”]
[Attachment]
tiflow version GitHub - pingcap/tiflow: This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB) v0.0.0-20220714025705-c7e31d529131
|
username: