Issues with Syncing VARBINARY and MEDIUMBLOB to Kafka Using TiCDC

translator_bot · June 22, 2024, 6:45am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC同步varbinary以及mediumblob至kafka的问题

| username: TIDB救我狗命

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 6.0.1
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]
The data written to Kafka is garbled.

translator_bot · June 22, 2024, 6:45am

| username: TIDB救我狗命 | Original post link

The message of Kafka is a string, it seems to be directly cast from binary. Is there any solution?

translator_bot · June 22, 2024, 6:45am

| username: nongfushanquan | Original post link

What protocol are you using? Is it canal-json? We have a documentation PR describing this issue, but it hasn’t been merged yet.

github.com/pingcap/docs

Documenting binary encoding used with Canal-JSON

pingcap:master ← benmeadowcroft:master

opened 04:24PM - 09 Jun 23 UTC

benmeadowcroft

+39 -0

### First-time contributors' checklist - [x] I've signed [**Contributor Licen…se Agreement**](https://cla-assistant.io/pingcap/docs) that's required for repo owners to accept my contribution. ### What is changed, added or deleted? (Required) I added documentation that describes how the binary data found in BINARY, VARBINARY, and BLOB columns is encoded when using the Canal-JSON format. ### Which TiDB version(s) do your changes apply to? (Required) - [x] master (the latest development version) - [ ] v7.2 (TiDB 7.2 versions) - [ ] v7.1 (TiDB 7.1 versions) - [ ] v7.0 (TiDB 7.0 versions) - [ ] v6.6 (TiDB 6.6 versions) - [ ] v6.5 (TiDB 6.5 versions) - [ ] v6.1 (TiDB 6.1 versions) - [ ] v5.4 (TiDB 5.4 versions) - [ ] v5.3 (TiDB 5.3 versions) - [ ] v5.2 (TiDB 5.2 versions) - [ ] v5.1 (TiDB 5.1 versions) - [ ] v5.0 (TiDB 5.0 versions) ### What is the related PR or file link(s)? None - This PR is translated from: N/A - Other reference link(s): N/A ### Do your changes match any of the following descriptions? - [ ] Delete files - [ ] Change aliases - [ ] Need modification after applied to another branch - [ ] Might cause conflicts after applied to another branch

translator_bot · June 22, 2024, 6:45am

| username: TIDB救我狗命 | Original post link

I am using the protobuf protocol.

translator_bot · June 22, 2024, 6:45am

| username: redgame | Original post link

You can try using a byte array to represent the message.

translator_bot · June 22, 2024, 6:45am

| username: Anna | Original post link

character-set = “auto”

Specifies the character set of the source data file. Lightning will convert the source file from the specified character set to UTF-8 encoding during the import process.

This configuration item is currently only used to specify the character set of CSV files. The following options are supported:

- utf8mb4: The source data file uses UTF-8 encoding.

- GB18030: The source data file uses GB-18030 encoding.

- GBK: The source data file uses GBK encoding (GBK encoding is an extension of the GB-2312 character set, also known as Code Page 936).

- binary: Do not attempt to convert encoding (default).

Leaving this configuration empty will default to “binary”, meaning no attempt to convert encoding.

It is important to note that Lightning does not make assumptions about the character set of the source data file and will only transcode and import data based on this configuration.

If the character set setting does not match the actual encoding of the source data file, it may result in import failure, missing data, or garbled data.