TiCDC sync to Kafka binary type is garbled

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc 同步到kafak binary类型是乱码

| username: TiDBer_LEF0OGCL

【TiDB Usage Environment】Production\Test Environment\POC
【TiDB Version】5.3.1
【Encountered Problem】
The hash value is stored in the database as binary(32). When ticdc synchronizes to Kafka in canal-json format, the message is garbled.
【Reproduction Path】What operations were performed to encounter the problem
【Problem Phenomenon and Impact】

image
image1380×42 19.9 KB

Database display
image
image1060×52 9.46 KB

【Attachments】

  • Relevant logs, configuration files, Grafana monitoring (https://metricstool.pingcap.com/)
  • TiUP Cluster Display information
  • TiUP Cluster Edit config information
  • TiDB-Overview monitoring
  • Corresponding module’s Grafana monitoring (such as BR, TiDB-binlog, TiCDC, etc.)
  • Corresponding module logs (including logs one hour before and after the issue)

If the question is related to performance optimization or fault troubleshooting, please download the script and run it. Please select all and copy-paste the terminal output results and upload them.

2
| username: xfworld | Original post link

This is not garbled text; it should have been escaped.

Looking at the logs you provided: repo_id is also like this, you can try reverse escaping it.

image
image797×813 24.2 KB

3
| username: xiaohetao | Original post link

:+1::+1::+1:

4
| username: xiaohetao | Original post link

Also learned a bit.

5
| username: TiDBer_LEF0OGCL | Original post link

It doesn’t seem to be this one. I’m using the canal-json protocol. The unescaping isn’t working well.

6
| username: Min_Chen | Original post link

Known issue, refer to Can not deserialize the binary/varbinary type data with canal-json format. · Issue #7058 · pingcap/tiflow · GitHub.
You can upgrade to 5.4.1 and test it.

7
| username: LingJin | Original post link

Related PRs

Reference implementation of canal-json decoder:
tiflow/canal_json_decoder.go at master · pingcap/tiflow (github.com)

8
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.