How to handle excessively large single fields when synchronizing data with DM?

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用dm同步数据时单字段太大如何处理?

| username: 宸凡_22

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]

restore table bx_azgl.img_base64 failed: Error 8025: entry too large, the max entry size is 6291456, the size of data is 7288764

| username: 小龙虾爱大龙虾 | Original post link

This is not because the table is too large, but because the row data exceeds TiDB’s single row 6M limit. Although this can be modified, it is not recommended as performance will not be good with such large rows. Check the table structure that is causing this error to see if it contains any large fields.

| username: 宸凡_22 | Original post link

This is the base64 code of the image stored in the table field.

| username: 小龙虾爱大龙虾 | Original post link

Not recommended to store images :joy_cat:

| username: Jellybean | Original post link

The error is obvious, the size of a single row of data exceeds the limit. To ensure the efficiency of distributed transactions, there is this hard limit.

The solution is to split the content of different columns into different tables so that the size of a single row meets the limit requirements.

| username: 宸凡_22 | Original post link

It’s not possible to split the columns. This table has only two columns: one for the ID and one for the value.

| username: zhaokede | Original post link

It is better to store files and images in a small file system. Only store key information such as storage paths in the database.

| username: 像风一样的男子 | Original post link

The performance of storing images in a database is generally not high.

| username: kelvin | Original post link

It is recommended that databases should avoid storing images as much as possible.

| username: dba远航 | Original post link

This is a mandatory system requirement, and it seems it cannot be changed.

| username: Hacker007 | Original post link

Indeed, there’s no way around it. If this table is not used in TiDB, you can filter out this table.

| username: Sunward | Original post link

You can only change the configuration.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.