Error initializing checkpoint during DM migration

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: DM迁移时报错init checkpoint error

| username: Tiny

[TiDB Usage Environment] Production Environment / Poc
[TiDB Version] v8.1.0
[Encountered Problem: Problem Phenomenon and Impact]
DM task error paused, error message:

[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1

Check the logs under the corresponding DM worker, as follows:

[2024/06/05 09:05:10.667 +08:00] [INFO] [import.go:903] ["restore all schema completed"] [task=saas_to_tidb] [unit=lightning-load] [takeTime=285.991013ms] []
[2024/06/05 09:05:10.668 +08:00] [INFO] [version.go:440] ["detect server version"] [type=TiDB] [version=8.1.0]
[2024/06/05 09:05:11.959 +08:00] [INFO] [tidb.go:295] ["skipping check requirements for tidb backend"] [task=saas_to_tidb] [unit=lightning-load]
[2024/06/05 09:05:11.959 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=]
[2024/06/05 09:05:12.044 +08:00] [WARN] [util.go:201] ["insert checkpoints failed with no retry"] [task=saas_to_tidb] [unit=lightning-load] [retryCnt=0] [error="Error 1406 (22001): Data too long for column 'source_dir' at row 1"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [import.go:569] ["run failed"] [task=saas_to_tidb] [unit=lightning-load] [step=3] [error="[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [import.go:579] ["the whole procedure failed"] [task=saas_to_tidb] [unit=lightning-load] [takeTime=1.662899038s] [error="[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [lightning.go:484] ["failed to run lightning"] [task=saas_to_tidb] [unit=lightning-load] [error="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"] [errorVerbose="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:337\ngithub.com/pingcap/tiflow/dm/loader.convertLightningError\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:318\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:299\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:475\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:543\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Resume\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:611\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [lightning.go:544] ["process error"] [task=saas_to_tidb] [unit=lightning-load] [error="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"] [errorVerbose="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:337\ngithub.com/pingcap/tiflow/dm/loader.convertLightningError\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:318\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:299\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:475\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:543\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Resume\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:611\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/06/05 09:05:12.044 +08:00] [INFO] [lightning.go:632] ["progress status of lightning"] [task=saas_to_tidb] [unit=lightning-load] [finished_bytes=0] [total_bytes=2595403032923] [progress="0.00 %"] ["current speed (bytes / seconds)"=0]
[2024/06/05 09:05:12.044 +08:00] [INFO] [lightning.go:556] ["lightning load end"] [task=saas_to_tidb] [unit=lightning-load] [IsCanceled=false] [finished_bytes=0] [total_bytes=2595403032923] [progress="0.00 %"]
[2024/06/05 09:05:12.044 +08:00] [INFO] [lightning.go:632] ["progress status of lightning"] [task=saas_to_tidb] [unit=lightning-load] [finished_bytes=0] [total_bytes=2595403032923] [progress="0.00 %"] ["current speed (bytes / seconds)"=0]
[2024/06/05 09:05:12.044 +08:00] [INFO] [subtask.go:338] ["unit process returned"] [subtask=saas_to_tidb] [unit=Load] [stage=Paused] [status="{\"totalBytes\":2595403032923,\"progress\":\"0.00 %\",\"metaBinlog\":\"(mysql-bin.200129, 56390260)\",\"metaBinlogGTID\":\"06aea9a9-3254-11ee-84d6-b8cef6abafc4:1-9866426949,073f658f-9cfb-11ed-ad7a-b8cef6286318:1-73380394,0b060c54-9732-11ed-84f9-b8cef6204434:1-11950259,1138f250-84b7-11ed-9bad-b8cef6204434:1-44644505,1c044756-822e-11ed-b901-b8cef620662c:1-78939241,2a66430c-9f55-11ed-8ca9-b8cef620662c:1-371343241,4526add7-861f-11ed-bbca-b8cef620662c:1-5001207,48634626-1a20-11ed-abfd-b8cef6204434:1-1366722044,4e92d873-ec4e-11ec-a38f-b8cef620662c:1-253880812,54760216-437b-11ed-9719-b8cef6ab7ca4:1-801633677,69a4c9e7-5906-11ed-bcae-b8cef6205b35:1-391191583,79da58f2-aa65-11ed-867a-b8cef6286318:1-129656734,7b67f9c5-aee1-11ed-af56-b8cef62080dc:1-4288664626,861e55bf-9460-11ed-8ae7-b8cef6204434:1-38063549,86e0231b-9595-11ed-ad05-b8cef620662c:1-54130804,a9715f72-9787-11ed-94fc-b8cef620662c:1-172830361,af2e25f1-919a-11ed-a60a-b8cef620662c:1-26121580:26121582-26121584:26121587:26121589-26121651,b6b31d19-87ae-11ed-b33e-b8cef620662c:1-41855135,bc4ee0b4-8641-11ed-b7b7-b8cef6204434:1-51934417,bdd89ef1-0d88-11ed-a756-b8cef62080e4:1-290031999,c0d30793-6217-11ed-b390-b8cef6202f94:1-1299616952,c6484d6f-8df5-11ed-8caf-b8cef6204434:1-131050974:131050976-131051192:131051194-131051305:131051307-131051359,c78369a2-8bb4-11ed-92ee-b8cef620662c:1-79347384,cab28f12-9317-11ed-92bc-b8cef620662c:1-43069687,ce72cd92-9252-11ed-afa4-b8cef6204434:1-26135774:26135776-26135783,fd014043-88e3-11ed-ac48-b8cef6204434:1-87597635\"}"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [subtask.go:359] ["unit process error"] [subtask=saas_to_tidb] [unit=Load] ["error information"="ErrCode:34019 ErrClass:\"load-unit\" ErrScope:\"internal\" ErrLevel:\"high\" RawCause:\"[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1\" "]
[2024/06/05 09:05:12.044 +08:00] [INFO] [subtask.go:361] [paused] [subtask=saas_to_tidb] [unit=Load]
[2024/06/05 09:05:13.066 +08:00] [WARN] [task_checker.go:352] ["task can't auto resume"] [component="task checker"] [task=saas_to_tidb] ["paused duration"=9h30m19.999908942s]
[2024/06/05 09:05:18.066 +08:00] [WARN] [task_checker.go:352] ["task can't auto resume"] [component="task checker"] [task=saas_to_tidb] ["paused duration"=9h30m24.999698406s]
[2024/06/05 09:05:23.067 +08:00] [WARN] [task_checker.go:352] ["task can't auto resume"] [component="task checker"] [task=saas_to_tidb] ["paused duration"=9h30m30.00058815s]
| username: xfworld | Original post link

Check the length.

| username: Tiny | Original post link

In the target database, there is a temporary database saas_to_tidb_1107585871_tidb_lightning_checkpoint, which contains a table task_v2. I increased the length of source_dir from varchar 256 to 1024, and it worked fine. Previously, I was looking for the source_dir table in dm_meta and lightning_task_info.

| username: zhaokede | Original post link

The error is obvious and easy to fix.

| username: WinterLiu | Original post link

This error is quite obvious and easy to handle.

| username: TiDBer_QYr0vohO | Original post link

Data too long for column ‘source_dir’

| username: jiayou64 | Original post link

:clap: Glad it’s resolved :call_me_hand: