Does TiDB database have plans for conditional filtering for data synchronization and CDC to Kafka consumption in non-primary key order?

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb数据库有没有规划条件筛选做数据同步。及cdc到kafka按非主键顺序消费

| username: miya

Requirement Feedback
[Problem Scenario Involved in the Requirement]
Recently, we have been aligning with the core function of our related housing system, specifically the脱O方案. This plan involves our core functionality. The related party is a table with over 200 fields, but we only need a subset of these fields and only a portion of the data. Moreover, since our system is directly user-facing, it requires this part of the data to be synchronized in real-time to avoid users seeing inconsistent data. Therefore, we adopted the CDC to Kafka solution and performed column pruning. However, it still does not meet the requirements. Due to the large volume of data, full synchronization cannot meet the requirements, so we need to filter the data. After researching, we found that TiDB does not provide such functionality. Additionally, due to the business scenario, we need the data to be consumed in Kafka in the order of customer numbers, which are not the primary key of the table. We have not found a configuration method for this.
[Expected Requirement Behavior]
Filter the data while synchronizing to meet real-time synchronization requirements.
[Alternative Solutions]
None at the moment.
[Background Information]
Users can view customer data in real-time, providing precise services to customers.

| username: tidb菜鸟一只 | Original post link

Can’t filter while consuming in Kafka?

| username: miya | Original post link

Kafka can be used for filtering, but it is not suitable for this scenario. We need to query the database for verification, and if we need to perform hundreds of millions of queries in a day, the performance impact would be unacceptable.