How to Limit TiCDC Memory Usage

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何限制TiCDC内存使用上限

| username: TiDBer_yyy

[TiDB Usage Environment] Production Environment
[TiDB Version] 7.1.0

[Encountered Issue: Phenomenon and Impact]

Recently, there have been frequent OOM (Out of Memory) occurrences in the TiFlash and TiDB components. We need to investigate the memory limit control of the TiCDC component to prevent OOM from causing widespread failures.

Investigation Progress:

  1. Checked the official Chinese documentation and only found per-table-memory-quota which limits “single table memory”?
  2. Found MaxMemoryPercentage and PerTableMemoryQuota parameters on GitHub, but they are marked as Deprecated: we don't use this field anymore. Are these two parameters no longer in use? What do these two parameters mean?
    tiflow/pkg/config/server_config.go at release-7.1 · pingcap/tiflow · GitHub
  3. The max-memory-percentage in the Chinese and English documentation is different (the Chinese version does not have this parameter, but the English version does). Is the max-memory-percentage parameter effective, and which documentation should be followed?
    TiCDC Server Configurations | PingCAP Docs
  • Additional Learning
    a. Describes that pull base sink can store data to local disk in LSM-tree manner through pull method, and set a memory threshold for each table to jointly reduce memory usage and reduce the risk of OOM.
    Lessons learned from TiCDC | Amyangfei's Blog
| username: 像风一样的男子 | Original post link

In the task Changefeed, you can set the upper limit of the memory quota in the Capture Server. For the excess usage, it will be prioritized for reclamation by the Go runtime during operation. The default value is 1073741824, which is 1 GB.
memory-quota = 1073741824

| username: Daniel-W | Original post link

The sorter.max-memory-consumption is used to set the total memory usage for CDC. You can check the CDC configuration documentation for parameters related to memory limits.

| username: TiDBer_yyy | Original post link

Understood. Besides this, are there any overall upper limits like max_memory_usage_for_all_queries for components such as TiFlash and TiDB?

Based on practical usage, setting the memory limit for each Changefeed has two issues:

  1. It can only limit the memory of a single task, and updating the memory for all tasks requires updating each one individually.
  2. If the machine has 32G of memory, using 80% would allow for configuring up to 25.6 tasks per machine, but in reality, it might only use 5-6G of memory. From a management perspective, more tasks can be added for synchronization; however, from a frontline operations perspective, this poses a risk. Ultimately, there is still a desire for a method to limit the overall memory usage.
| username: 像风一样的男子 | Original post link

Why does the English version have this configuration, but the Chinese documentation does not?

| username: TiDBer_yyy | Original post link

According to the GitHub repository description, the parameters max-memory-consumption and max-memory-percentage are no longer used.

It seems that the parameter gc-tuner-memory-threshold is the threshold for triggering GC.
Setting gc-tuner-memory-threshold should be effective.

server.run()server.prepare()server.setMemoryLimit()setMemoryLimitgctuner.Tuning(conf.GcTunerMemoryThreshold)

| username: TiDBer_yyy | Original post link

It seems that it is no longer needed, but this parameter is still present in the code.

| username: Daniel-W | Original post link

6.5 works and takes effect, but I haven’t tested higher versions.

| username: TiDBer_小阿飞 | Original post link

On the TiKV side, there is a CDC component used to track changes in TiKV and send data to downstream TiCDC. The size of the old value cache can be controlled by the parameter cdc.old-value-cache-memory-quota, and the memory size occupied by the CDC change events waiting to be sent in TiKV can be controlled by cdc.sink-memory-quota.

| username: 像风一样的男子 | Original post link

Confirmed that this parameter is not effective, and it will be removed from the documentation later.

| username: TiDBer_yyy | Original post link

Is the gc-tuner-memory-threshold parameter effective?

| username: 像风一样的男子 | Original post link

If it is not found in the documentation, then it is invalid.

| username: TiDBer_yyy | Original post link

Okay, I think the constant assigned during initialization is probably invalid.

| username: Fly-bird | Original post link

The image is not visible. Please provide the text you need translated.

| username: TiDBer_yyy | Original post link

It seems that this parameter is invalid in version 7.

| username: Kongdom | Original post link

Are you referring to this?

| username: Daniel-W | Original post link

May I ask, is there a specific version from which it stops being effective?

| username: zxgaa | Original post link

I’m using version 6.1, tried all kinds of parameters, but none of them worked. At most, I can use up to 350GB of memory.

| username: TiDBer_yyy | Original post link

Yes, these two points should be able to indirectly limit memory usage, but they cannot directly limit memory usage at the server level.

| username: lilin90 | Original post link

Thank you for discovering and reporting the inconsistency between the Chinese and English documents. The cdc server configuration file section in the Chinese TiCDC Server Configuration and the English TiCDC Server Configurations has been updated to reflect consistent content.