After upgrading TiDB from 5.4.2 to 6.1.2, the text field frequently reports errors due to excessively large data

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb5.4.2升级6.1.2之后 text字段频繁出现数据对于字段过大的报错 。

| username: heming

[TiDB Usage Environment] Production Environment / Test / Poc
Production
[TiDB Version]
6.1.2
[Reproduction Path] What operations were performed when the issue occurred
After upgrading from 5.4.2 to 6.1.2, this error frequently occurs.
Frequent “Data too long for column ‘video_material’ at row” errors: (also errors for the ‘image_material’ field)

CREATE TABLE `Synads_Tt_Qc_Creative` (
  `creative_id` bigint(63) NOT NULL COMMENT 'Creative ID',
  `ad_id` bigint(63) DEFAULT NULL COMMENT 'Plan ID',
  `advertiser_id` bigint(63) DEFAULT NULL COMMENT 'Advertiser ID',
  `status` varchar(500) DEFAULT NULL COMMENT 'Plan delivery status',
  `opt_status` varchar(500) DEFAULT NULL COMMENT 'Plan operation status',
  `image_mode` varchar(500) DEFAULT NULL COMMENT 'Creative material type',
  `creative_create_time` varchar(100) DEFAULT NULL COMMENT 'Creative creation time, format: yyyy-mm-dd HH:MM:SS',
  `creative_modify_time` varchar(100) DEFAULT NULL COMMENT 'Creative modification time, format: yyyy-mm-dd HH:MM:SS',
  `video_material` text DEFAULT NULL COMMENT 'Video material information',
  `image_material` text DEFAULT NULL COMMENT 'Image material information',
  `title_material` text DEFAULT NULL COMMENT 'Title material information',
  `promotion_card_material` varchar(1500) DEFAULT NULL COMMENT 'Promotion card information',
  `Synch_Create_Date` varchar(25) NOT NULL COMMENT 'Format: yyyy-mm-dd',
  `lab_ad_type` varchar(255) DEFAULT NULL COMMENT 'Managed plan type, NOT_LAB_AD: non-managed plan, LAB_AD: managed plan',
  `Synch_Create_Hour` int(11) NOT NULL COMMENT 'Hour value range: 0~23',
  `Synch_Time` char(19) DEFAULT NULL,
  `marketing_goal` varchar(255) DEFAULT NULL COMMENT 'Marketing goal, allowed values: VIDEO_PROM_GOODS short video goods, LIVE_PROM_GOODS live goods',
  `creative_material_mode` varchar(60) DEFAULT NULL COMMENT 'Creative presentation mode, CUSTOM_CREATIVE: custom creative, PROGRAMMATIC_CREATIVE: programmatic creative',
  PRIMARY KEY (`creative_id`) /*T![clustered_index] CLUSTERED */,
  KEY `creative_create_time` (`creative_create_time`),
  KEY `creative_modify_time` (`creative_modify_time`),
  KEY `Synch_Create_Date` (`Synch_Create_Date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin

[Encountered Issues: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
2022/11/17 11:29:25.359 +08:00] [INFO] [conn.go:1149] [command dispatched failed] [conn=985285563792585879] [connInfo=id:985285563792585879, addr:!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQ\Temp%W@GJ$ACOF(TYDYECOKVDYB.png)192.168.6.29:64675 status:10, collation:utf8mb4_general_ci, user:dspwrite] [command=Execute] [status=inTxn:0, au[types:1406]Data too long for column ‘video_material’ at row 1!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQ\Temp%W@GJ$ACOF(TYDYECOKVDYB.png)ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQTempSys[5UQ[BL(6~BS2JV6W}N6[%S.png)github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQ\Temp%W@GJ$ACOF(TYDYECOKVDYB.png)ngithub.com/pingcap/errors.(*Error).GenWithStack\n\t/go/pkg/mod/!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQTempSys[5UQ[BL(6~BS2JV6W}N6[%S.png)github.com/pingcap/errors@v0.11.5-0.2021122
Path:/data/tidb/tidblog/tidb.log
2022/11/17 11:29:02.969 +08:00] [INFO] [conn.go:1149] [command dispatched failed] [conn=985285563792585877] [connInfo=id:985285563792585877, addr:!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQ\Temp%W@GJ$ACOF(TYDYECOKVDYB.png)192.168.6.29:46303 status:10, collation:utf8mb4_general_ci, user:dspwrite] [command=Execute] [status=inTxn:0, au[types:1406]Data too long for column ‘image_material’ at row 1!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQ\Temp%W@GJ$ACOF(TYDYECOKVDYB.png)ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQTempSys[5UQ[BL(6~BS2JV6W}N6[%S.png)github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQ\Temp%W@GJ$ACOF(TYDYECOKVDYB.png)ngithub.com/pingcap/errors.(*Error).GenWithStack\n\t/go/pkg/mod/!(file:///C:\Users\heming\AppData\Roaming\Tencent\QQTempSys[5UQ[BL(6~BS2JV6W}N6[%S.png)github.com/pingcap/errors@v0.11.5-0.2021122


[Attachments]:
- Related logs
- Configuration files
- Grafana monitoring (https://metricstool.pingcap.com/)

If the question is related to **performance optimization or fault troubleshooting**, please download the <a href="/uploads/short-url/uGisshjxFnxx1KgpFOYbfeZjsc6" download="info_gathering.py">script</a> and run it. Please **select all** and copy-paste the terminal output results and upload them.
| username: db_user | Original post link

You can check if the character set is consistent with the old version and if the length is really over the limit.


Also, check if the sql_mode is consistent with the old version and if both are in strict mode.

| username: heming | Original post link

select @@sql_mode; # Consistent.
| @@sql_mode |
|STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |

I don’t know if it has anything to do with the character set ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin

| username: heming | Original post link

Status:10, collation:utf8mb4_general_ci. Not sure what this situation is. Could it be related to the character set?

| username: db_user | Original post link

It may have something to do with the character set. For example, the number of bytes occupied by a single Chinese character in utf8 and utf8mb4 is inconsistent. You can compare the corresponding length() and char_length() to see if the results are consistent and if they really reach the maximum value.

| username: heming | Original post link

The error statement looks like the data is not large. It should not be a real data length issue. Tried longtext but it didn’t work either.

INSERT INTO `Synads_Tt_Qc_Creative` (`creative_id`, `ad_id`, `advertiser_id`, `status`, `opt_status`, `image_mode`, `creative_create_time`, `creative_modify_time`, `video_material`, `image_material`, `title_material`, `promotion_card_material`, `Synch_Create_Date`, `Synch_Create_Hour`, `lab_ad_type`, `Synch_Time`, `creative_material_mode`, `marketing_goal`) VALUES (?, ?, ?, ?, ?, ?, ?, ?, IFNULL(?, ?), IFNULL(?, ?), IFNULL(?, ?), ?, ?, ?, ?, NOW(), ?, ?) ON DUPLICATE KEY UPDATE status = ?, opt_status = ?, image_mode = ?, video_material = ?, creative_modify_time = ?, video_material = IFNULL(?, ?), image_material = IFNULL(?, ?), title_material = IFNULL(?, ?), promotion_card_material = ?, Synch_Create_Hour = ?, lab_ad_type = ?, Synch_Time = NOW(), creative_material_mode = ?, marketing_goal = ?
[arguments: (1733190621, 173318500, 171040419, "DELETE", "DISABLE", "VIDEO_LARGE", "2022-05-19 02:43:05", "2022-11-17 16:09:48", "[{\"aweme_item_id\":0,\"image_mode\":null,\"is_auto_generate\":0,\"video_cover_id\":\"tos-cn-v-c9e10a/537d35a58d2f41ecab2ac2\",\"video_id\":\"v02c8eg10000c9sjpq3c77\"}]", "{\"aweme_item_id\":0,\"is_auto_generate\":0,\"video_cover_id\":\"tos-cn-v-c9e10a/537d35a58d2f41e、\",\"video_id\":\"v02c8eg10000c9sjpq3c7\"}", NULL, NULL, "[{\"dynamic_words\":[],\"title\":\"The Valentine's Day gift box from Flyco is now available, hurry to the live room to grab it! #GiftForBoyfriend #MenMustHave #Flyco #Shaving #Razor\"}]", "{\"dynamic_words\":[],\"title\":\"The Valentine's Day gift box from Flyco is now available, hurry to the live room to grab it! #GiftForBoyfriend #MenMustHave #Flyco #Shaving #Razor\"}", NULL, "2022-05-19", "02", "NOT_LAB_AD", "PROGRAMMATIC_CREATIVE", "LIVE_PROM_GOODS", "DELETE", "DISABLE", "VIDEO_LARGE", "{\"aweme_item_id\":0,\"is_auto_generate\":0,\"video_cover_id\":\"tos-cn-v-c9e10a/537d35a58d2f41ecab2ac\",\"video_id\":\"v02c8eg10000c9sjpq3c77\"}", "2022-11-17 16:09:48", "[{\"aweme_item_id\":0,\"image_mode\":null,\"is_auto_generate\":0,\"video_cover_id\":\"tos-cn-v-c9e10a/537d35a58d2f\",\"video_id\":\"v02c8eg1000、\"}]", "{\"aweme_item_id\":0,\"is_auto_generate\":0,\"video_cover_id\":\"tos-cn-v-c9e10a/537d35a58d2f41eca\",\"video_id\":\"v02c8eg100\"}", NULL, NULL, "[{\"dynamic_words\":[],\"title\":\"The Valentine's Day gift box from Flyco is now available, hurry to the live room to grab it! #GiftForBoyfriend #MenMustHave #Flyco #Shaving #Razor\"}]", "{\"dynamic_words\":[],\"title\":\"The Valentine's Day gift box from Flyco is now available, hurry to the live room to grab it! #GiftForBoyfriend #MenMustHave #Flyco #Shaving #Razor\"}", NULL, "02", "NOT_LAB_AD", "PROGRAMMATIC_CREATIVE", "LIVE_PROM_GOODS")]
| username: foxchan | Original post link

Started a standalone TiDB version 5.4.3, no errors reported.

| username: heming | Original post link

Please help check which part of TiDB 6.1.2 has an issue. The same backend TiKV 6.1.2 does not report an error with the frontend TiDB 5.4.2 and 5.4.3.

| username: db_user | Original post link

Can you provide a standard SQL for this? I want to reproduce it on my end.

| username: heming | Original post link

It seems to be fine to execute like this.
update Synads_Tt_Qc_Creative set video_material=‘[{“dynamic_words”:,“title”:“The Flyco Valentine’s Day gift box is now available, hurry to the live broadcast room to grab it! #GiftForBoyfriend #MenEssentials #Flyco #Shaving #Razor”}]’ where creative_id=1733190621287435;

| username: db_user | Original post link

It seems to be this bug, affecting version 6.1. Try removing ifnull and test it.

| username: heming | Original post link

Can this bug be fixed as soon as possible?

| username: db_user | Original post link

This has already been closed, the bug has been fixed.

| username: heming | Original post link

I have tried it, and it is indeed an issue with ifnull. Has the bug fix version been merged into any release version?

| username: db_user | Original post link

The issue was resolved in release-6.1.

| username: heming | Original post link

In version 6.1.2, the index key from the original table cannot be recognized in the view, and using force index reports “key not found”. It works normally with TiDB Server version 5.4.3.

| username: Billmay表妹 | Original post link

Post a new thread for the new issue to avoid being overlooked.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.