Will TiFlash consider supporting simple data transformations, adding columns, and other functions similar to OGG?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash是否会考虑支持简单的数据转换、增加列等类似OGG的一些功能?

| username: alfred

【TiDB Usage Environment】Production Environment or Testing Environment or POC
【TiDB Version】
【Encountered Problem】
【Reproduction Path】What operations were performed that led to the problem
【Problem Phenomenon and Impact】

【Attachments】 Relevant logs and monitoring (https://metricstool.pingcap.com/)


For questions related to performance optimization or troubleshooting, please download and run the script. Please select all and copy-paste the terminal output results for upload.

| username: yilong | Original post link

  1. Oracle’s OGG is a log-based structured replication software.
  2. TiFlash is a key component of TiDB’s HTAP architecture. It is a columnar storage extension of TiKV, providing good isolation while also ensuring strong consistency. The columnar replicas are asynchronously replicated through the Raft Learner protocol, but during reads, they achieve Snapshot Isolation consistency level by coordinating Raft index checks with MVCC.

TiFlash primarily synchronizes data through the Raft protocol, and these features are currently not expected to be added during the synchronization process.

| username: alfred | Original post link

Yes, the replication principles are indeed different.

| username: alfred | Original post link

If you require some data conversion, it is indeed asking too much. :sweat_smile:

| username: forever | Original post link

This definitely won’t be added. TFlash column storage is mainly used for OLAP queries and MPP. The most important thing is to ensure data consistency with KV. If data transformation and processing are supported, it means the data is altered. So if the execution plan reaches TFlash, wouldn’t the result be incorrect?

| username: alfred | Original post link

TiFlash is similar to the ODS layer.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.