TiDB-v6.1 DM Synchronization Error

translator_bot · June 23, 2024, 10:21am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb-v6.1 dm同步报错

| username: Hacker_lB0K9iQY

【TiDB Usage Environment】POC
【TiDB Version】tidb 6.1
【Encountered Problem】
【Reproduction Path】Close DM
【Problem Phenomenon and Impact】DM cannot properly synchronize upstream data
TiDB DM error log is as follows:
[2022/07/11 15:52:04.215 +08:00] [INFO] [source_worker.go:630] [“OperateSubTask start”] [component=“worker controller”] [op=AutoResume] [task=eo_school_dm_task_test1]
[2022/07/11 15:52:04.215 +08:00] [INFO] [source_worker.go:653] [“auto_resume subtask”] [component=“worker controller”] [task=eo_school_dm_task_test1]
[2022/07/11 15:52:04.215 +08:00] [INFO] [subtask.go:638] [“resume with unit”] [subtask=eo_school_dm_task_test1] [unit=Sync]
[2022/07/11 15:52:04.215 +08:00] [INFO] [source_worker.go:658] [“OperateSubTask finished”] [component=“worker controller”] [op=AutoResume] [task=eo_school_dm_task_test1]
[2022/07/11 15:52:04.215 +08:00] [INFO] [task_checker.go:401] [“dispatch auto resume task”] [component=“task checker”] [task=eo_school_dm_task_test1]
[2022/07/11 15:52:04.563 +08:00] [INFO] [syncer.go:1531] [“received ungraceful exit ctx, exit now”] [task=eo_school_dm_task_test1] [unit=“binlog replication”]
[2022/07/11 15:52:04.563 +08:00] [INFO] [subtask.go:356] [“unit process returned”] [subtask=eo_school_dm_task_test1] [unit=Sync] [stage=Paused] [status=“{"syncerBinlog":"(mysql-bin.000014, 779599775)","syncerBinlogGtid":"44950f9d-e7b9-11ea-8cef-e4434bcbb70e:1-986313129,5e16e963-f828-11ec-8675-78ac44235198:1-3,c698a928-abde-11ea-99b7-e4434bcd09dc:1","binlogType":"remote"}”]
[2022/07/11 15:52:04.582 +08:00] [ERROR] [subtask.go:377] [“unit process error”] [subtask=eo_school_dm_task_test1] [unit=Sync] [“error information”="ErrCode:50000 ErrClass:"not-set" ErrScope:"not-set" ErrLevel:"high" Message:"tikv aborts txn: Value with size 6229211 exceeded ValueLogFileSize (4194304). Key:\
00000000 7b 22 69 64 22 3a 37 36 39 2c 22 74 79 70 65 22 |{\"id\":769,\"type\"|\
00000010 3a 36 30 2c 22 73 63 68 65 6d 61 5f 69 64 22 3a |:60,\"schema_id\":|\
00000020 37 30 2c 22 74 61 62 6c 65 5f 69 64 22 3a 30 2c |70,\"table_id\":0,|\
00000030 22 73 63 68 65 6d 61 5f 6e 61 6d 65 22 3a 22 65 |\"schema_name\":\"e|\
00000040 6f 5f 73 63 68 6f 6f 6c 22 2c 22 74 61 62 6c 65 |o_school\",\"table|\
00000050 5f 6e 61 6d 65 22 3a 22 22 2c 22 73 74 61 74 65 |name\":\"\",\"state|\
00000060 22 3a 38 2c 22 65 72 72 22 3a 6e 75 6c 6c 2c 22 |\":8,\"err\":null,\"|\
00000070 65 72 72 5f 63 6f 75 6e 74 22 3a 30 2c 22 72 6f |err_count\":0,\"ro|\
00000080 77 5f 63 6f 75 6e 74 22 3a 30 2c 22 72 61 77 5f |w_count\":0,\"raw|\
00000090 61 72 67 73 22 3a 5b 5b 7b 22 69 64 22 3a 37 32 |args\":[[{\"id\":72|\
000000a0 2c 22 6e 61 6d 65 22 3a 7b 22 4f 22 3a 22 43 6f |,\"name\":{\"O\":\"Co|\
000000b0 75 72 73 65 55 73 65 72 5f 30 30 32 34 22 2c 22 |urseUser_0024\",\"|\
000000c0 4c 22 3a 22 63 6f 75 72 73 65 75 73 65 72 5f 30 |L\":\"courseuser_0|\
000000d0 30 32 34 22 7d 2c 22 63 68 61 72 73 65 74 22 3a |024\"},\"charset\":|\
000000e0 22 75 74 66 38 6d 62 34 22 2c 22 63 6f 6c 6c 61 |\"utf8mb4\",\"colla|\
000000f0 74 65 22 3a 22 75 74 66 38 6d 62 34 5f 75 6e 69 |te\":\"utf8mb4_uni|\
00000100 63 6f 64 65 5f 63 69 22 2c 22 63 6f 6c 73 22 3a |code_ci\",\"cols\":|\
00000110 5b 7b 22 69 64 22 3a 31 2c 22 6e 61 6d 65 22 3a |[{\"id\":1,\"name\":|\
00000120 7b 22 4f 22 3a 22 43 6f 75 72 73 65 49 44 22 2c |{\"O\":\"CourseID\",|\
00000130 22 4c 22 3a 22 63 6f 75 72 73 65 69 64 22 7d 2c |\"L\":\"courseid\"},|\
00000140 22 6f 66 66 73 65 74 22 3a 30 2c 22 6f 72 69 67 |\"offset\":0,\"orig|\
00000150 69 6e 5f 64 65 66 61 75 6c 74 22 3a 6e 75 6c 6c |in_default\":null|\
00000160 2c 22 6f 72 69 67 69 6e 5f 64 65 66 61 75 6c 74 |,\"origin_default|\
00000170 5f 62 69 74 22 3a 6e 75 6c 6c 2c 22 64 65 66 61 |bit\":null,\"defa|\
00000180 75 6c 74 22 3a 6e 75 6c 6c 2c 22 64 65 66 61 75 |ult\":null,\"defau|\
00000190 6c 74 5f 62 69 74 22 3a 6e 75 6c 6c 2c 22 64 65 |lt_bit\":null,\"de|\
000001a0 66 61 75 6c 74 5f 69 73 5f 65 78 70 72 22 3a 66 |fault_is_expr\":f|\
000001b0 61 6c 73 65 2c 22 67 65 6e 65 72 61 74 65 64 5f |alse,\"generated|\
000001c0 65 78 70 72 5f 73 74 72 69 6e 67 22 3a 22 22 2c |expr_string\":\"\",|\
000001d0 22 67 65 6e 65 72 61 74 65 64 5f 73 74 6f 72 65 |generated_store|\
000001e0 64 22 3a 66 61 6c 73 65 2c 22 64 65 70 65 6e 64 |d\":false,\"depend|\
000001f0 65 6e 63 65 73 22 3a 6e 75 6c 6c 2c 22 74 79 70 |ences\":null,\"typ|\
00000200 65 22 3a 7b 22 54 70 22 3a 38 2c 22 46 6c 61 67 |e\":{\"Tp\":8,\"Flag|\
00000210 22 3a 34 31 33 31 2c 22 46 6c 65 6e 22 3a 32 30 |\":4131,\"Flen\":20|\
00000220 2c 22 44 65 63 69 6d 61 6c 22 3a 30 2c 22 43 68 |,\"Decimal\":0,\"Ch|\
00000230 61 72 73 65 74 22 3a 22 62 69 6e 61 72 79 22 2c |arset\":\"binary\",|\
00000240 22 43 6f 6c 6c 61 74 65 22 3a 22 62 69 6e 61 72 |\"Collate\":\"binar|\
00000250 79 22 2c 22 45 6c 65 6d 73 22 3a 6e 75 6c 6c 7d |y\",\"Elems\":null}|\
00000260 2c 22 73 74 61 74 65 22 3a 35 2c 22 63 6f 6d 6d |,\"state\":5,\"comm|\
00000270 65 6e 74 22 3a 22 e8 af be e7 a8 8b e5 94 af e4 |ent\":\"…|\
00000280 b8 80 e6 a0 87 e8 af 86 20 22 2c 22 68 69 64 64 |… \",\"hidd|\
00000290 65 6e 22 3a 66 61 6c 73 65 2c 22 63 68 61 6e 67 |en\":false,\"chang|\
000002a0 65 5f 73 74 61 74 65 5f 69 6e 66 6f 22 3a 6e 75 |e_state_info\":nu|\
000002b0 6c 6c 2c 22 76 65 72 73 69 6f 6e 22 3a 32 7d 2c |ll,\"version\":2},|\
000002c0 7b 22 69 64 22 3a 32 2c 22 6e 61 6d 65 22 3a 7b |{\"id\":2,\"name\":{|\
000002d0 22 4f 22 3a 22 55 49 44 22 2c 22 4c 22 3a 22 75 |\"O\":\"UID\",\"L\":\"u|\
000002e0 69 64 22 7d 2c 22 6f 66 66 73 65 74 22 3a 31 2c |id\"},\"offset\":1,|\
000002f0 22 6f 72 69 67 69 6e 5f 64 65 66 61 75 6c 74 22 |\"origin_default\"|\
00000300 3a 6e 75 6c 6c 2c 22 6f 72 69 67 69 6e 5f 64 65 |:null,\"origin_de|\
00000310 66 61 75 6c 74 5f 62 69 74 22 3a 6e 75 6c 6c 2c |fault_bit\":null,|\
00000320 22 64 65 66 61 75 6c 74 22 3a 6e 75 6c 6c 2c 22 |\"default\":null,\"|\
00000330 64 65 66 61 75 6c 74 5f 62 69 74 22 3a 6e 75 6c |default_bit\":nul|\
00000340 6c 2c 22 64 65 66 61 75 6c 74 5f 69 73 5f 65 78 |l,\"default_is_ex|\
00000350 70 72 22 3a 66 61 6c 73 65 2c 22 67 65 6e 65 72 |pr\":false,\"gener|\
00000360 61 74 65 64 5f 65 78 70 72 5f 73 74 72 69 6e 67 |ated_expr_string|\
00000370 22 3a 22 22 2c 22 67 65 6e 65 72 61 74 65 64 5f |\":\"\",\"generated_|\
00000380 73 74 6f 72 65 64 22 3a 66 61 6c 73 65 2c 22 64 |stored\":false,\"d|\
00000390 65 70 65 6e 64 65 6e 63 65 73 22 3a 6e 75 6c 6c |ependences\":null|\
000003a0 2c 22 74 79 70 65 22 3a 7b 22 54 70 22 3a 38 2c |,\"type\":{\"Tp\":8,|\
000003b0 22 46 6c 61 67 22 3a 34 31 33 39 2c 22 46 6c 65 |\"Flag\":4139,\"Fle|\
000003c0 6e 22 3a 32 30 2c 22 44 65 63 69 6d 61 6c 22 3

translator_bot · June 23, 2024, 10:21am

| username: Hacker007 | Original post link

A single row of data can only be up to 6M, remember that.

translator_bot · June 23, 2024, 10:21am

| username: lance6716 | Original post link

I am very sorry, this is a known issue with version 6.1 of DM. You can try splitting the synchronization task to include fewer tables.

github.com/pingcap/tiflow

the 'txn-entry-size-limit' config is not effective in DM

opened 04:59AM - 04 Jul 22 UTC

closed 09:51AM - 08 Jul 22 UTC

zwathz

type/bug severity/major area/dm affects-6.0 affects-6.1

### What did you do? https://asktug.com/t/topic/694876 TiDB config: ``` +-…-----+---------------------+--------------------------------------+-----------+ | Type | Instance | Name | Value | +------+---------------------+--------------------------------------+-----------+ | tidb | xxxxx:4000 | performance.txn-entry-size-limit | 125829120 | | tikv | xxxxx:20161 | raftstore.raft-entry-max-size | 128MiB | +------+---------------------+--------------------------------------+-----------+ ``` DM error: ``` "errors": [ { "ErrCode": 50000, "ErrClass": "not-set", "ErrScope": "not-set", "ErrLevel": "high", "Message": "[kv:8025]entry too large, the max entry size is 6291456, the size of data is 7890542", "RawCause": "", "Workaround": "" } ], ``` ### What did you expect to see? DM task work normally. ### What did you see instead? DM task get an error. ### Versions of the cluster DM version (run `dmctl -V` or `dm-worker -V` or `dm-master -V`): ``` tiup is checking updates for component dmctl ... Starting component `dmctl`: /home/admin/.tiup/components/dmctl/v6.1.0/dmctl/dmctl -V Release Version: v6.1.0 Git Commit Hash: 9f5e3cea2fae0a3dce6cc5ca79e3bcd654b8aa99 Git Branch: heads/refs/tags/v6.1.0 UTC Build Time: 2022-06-07 11:55:24 Go Version: go version go1.18.2 linux/amd64 ``` TiDB Version ``` MySQL [(none)]> SELECT tidb_version()\G *************************** 1. row *************************** tidb_version(): Release Version: v6.1.0 Edition: Community Git Commit Hash: https://github.com/pingcap/tidb/commit/1a89decdb192cbdce6a7b0020d71128bc964d30f Git Branch: heads/refs/tags/v6.1.0 UTC Build Time: 2022-06-05 05:15:11 GoVersion: go1.18.2 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false 1 row in set (0.00 sec) ``` ### current status of DM cluster (execute `query-status <task-name>` in dmctl) ```console (paste current status of DM cluster here) ```

If it still doesn’t work, could you please show the SHOW CREATE TABLE statement for the table that includes the CourseID and CourseUser_0024 columns?

translator_bot · June 23, 2024, 10:21am

| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.