Will TiDB support a compression mechanism similar to the Archive engine?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB会支持类似Archive引擎的压缩机制吗？

| username: fenghaojiang

Will TiDB support compression similar to the Archive engine?
From what I understand, TiDB’s architecture is abstracted into a B+Tree, abstracted for the InnoDB engine. Will there be an addition of engines like Archive in the future? Because it feels like TiDB’s storage cost is a bit high. :sweat_smile:

S86xLerSRE
S86xLerSRE1380×944 226 KB

2
| username: 张雨齐0720 | Original post link

Subsequent support for compression is expected, allowing for the configuration of compression mechanisms to enable compressed storage, though it will consume CPU resources. A neighboring competitor’s compression algorithm reduced storage costs by 40%. Since both use LSM Tree engines, it should definitely be possible.

3
| username: ddhe9527 | Original post link

The high storage cost is mainly due to multiple replicas. RocksDB has compression enabled by default, and if you feel the compression ratio is not sufficient, you can adjust the compression-per-level configuration.

4
| username: 张雨齐0720 | Original post link

compression-per-level

  • Default compression algorithm for each level.
  • Default value for defaultcf: [“no”, “no”, “lz4”, “lz4”, “lz4”, “zstd”, “zstd”]
  • Default value for writecf: [“no”, “no”, “lz4”, “lz4”, “lz4”, “zstd”, “zstd”]
  • Default value for lockcf: [“no”, “no”, “no”, “no”, “no”, “no”, “no”]

bottommost-level-compression

  • Sets the compression algorithm for the bottommost level. This setting will override the compression-per-level setting.
  • Since the bottommost level does not directly use the last compression algorithm in the compression-per-level array from the start of writing data into the LSM-tree, using bottommost-level-compression allows the bottommost level to use the most effective compression algorithm from the beginning.
  • If you do not want to set the compression algorithm for the bottommost level, you can set this configuration item to disable.
  • Default value: “zstd”
5
| username: fenghaojiang | Original post link

Which competitor are you referring to?

6
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.