Parsing Binlog Messages in Kafka

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 关于解析kafka中的binlog消息

| username: qufudcj

[TiDB Usage Environment] Test
[TiDB Version] 4.0.4
[Reproduction Path] Issues with Alibaba Cloud DTS after Kafka restart
[Encountered Problem: How to parse messages in Kafka]
[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]

Currently, Alibaba Cloud DTS is used to read Kafka’s binlog and send it to Alibaba Cloud ADB. However, a few days ago, a Kafka instance restarted. Although the service did not stop, Alibaba Cloud DTS crashed and failed to consume messages. It provided me with an offset, and I wanted to see if the messages in this offset could be skipped. But after consuming, I found that the data format could not be parsed…

I am not familiar with Go language and cannot use the driver tool mentioned in the official documentation. I hope to get an answer on how to parse this data using Python or Shell: b’\x08\x00\x10\x89\x80\x80\xeb\x94\xe3\xc2\x89\x06\x1a\xca\x02\n\xc7\x02\n\nuat_xxljob\x12\rxxl_job_group\x1a\x0b\n\x02id\x12\x03int\x18\x01\x1a\x15\n\x08app_name\x12\x07varchar\x18\x00\x1a\x12\n\x05title\x12\x07varchar\x18\x00\x1a\x19\n\x0caddress_type\x12\x07tinyint\x18\x00\x1a\x16\n\x0caddress_list\x12\x04text\x18\x00\x1a\x19\n\x0bupdate_time\x12\x08datetime\x18\x00"\x94\x01\x08\x01\x12G\n\x04\x10\xb2\xea\x01\n\x0c2\ngoods-task\n\x122\x10\xe7\x94\x9f\xe4\xba\xa7-\xe5\x95\x86\xe5\x93\x81JOB\n\x02\x10\x00\n\x02\x08\x01\n\x152\x132022-11-29 17:44:48\x1aG\n\x04\x10\xb2\xea\x01\n\x0c2\ngoods-task\n\x122\x10\xe7\x94\x9f\xe4\xba\xa7-\xe5\x95\x86\xe5\x93\x81JOB\n\x02\x10\x00\n\x02\x08\x01\n\x152\x132022-11-29 17:44:18*\r\n\x07PRIMARY\x12\x02id’

2
| username: xfworld | Original post link

The official protobuf file is provided:

You can use this file to generate the parsing format. If you know Python, you can generate an API and data format that Python can manipulate.

Then, the code you posted can be decoded.

If you have a friend who understands Java, you can ask for help:

Here is a kafkaReader example, you can modify it accordingly and use it.

3
| username: qufudcj | Original post link

If I modify the KafkaAddr, offset, and topic inside, can I directly run this Go script to solve the problem?

4
| username: xfworld | Original post link

The parsed result is also the protobuf definition of binlog. It is uncertain whether it can meet your requirements, but you can give it a try.

5
| username: qufudcj | Original post link

Thank you, it’s resolved. I had a developer colleague make some changes to this kafkareader.

6
| username: xfworld | Original post link

Okay :+1:

7
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.