Setting the Total Memory for a TiDB Cluster: tidb_server_memory_limit

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb集群的总内存设置tidb_server_memory_limit

| username: TiDBer_rYOSh9JN

Version: V6.5.0
tidb_server_memory_limit
Is this parameter set for the total memory usage of the TiDB cluster or for a single machine’s memory?
How to configure the total memory usage of the TiDB cluster, including TiDB, TiKV, PD, TiFlash, etc.?

| username: Miracle | Original post link

What version is TiDB?

| username: TiDBer_小阿飞 | Original post link

tidb_server_memory_limit && tidb_server_memory_limit_gc_trigger

tidb_server_memory_limit controls the memory limit for tidb-server at runtime. When the process memory exceeds tidb_server_memory_limit * tidb_server_memory_limit_gc_trigger, it triggers Go GC to release memory.

You can set tidb_server_memory_limit as a percentage or a specific memory size. When set as a percentage, it means the memory limit is total memory * percentage. When set as a specific memory size, the units can be KB/MB/GB/TB or byte number.

tidb_server_memory_limit can be set to 0 but cannot be set to a negative number. Setting this variable to 0 disables the Go runtime’s MemoryLimit feature, effectively turning off tidb-server’s MemoryTuner feature.

MySQL [(none)]> set global tidb_server_memory_limit = 1GB;
Query OK, 0 rows affected (0.01 sec)

MySQL [(none)]> set global tidb_server_memory_limit = “80%”;
Query OK, 0 rows affected (0.01 sec)

The code to set tidb_server_memory_limit is as follows. The parseMemoryLimit function parses the memory limit to be set and finally stores it in the global variable memory.ServerMemoryLimit.

// Global variable to store memory limit
var ServerMemoryLimit            = atomicutil.NewUint64(0)

// Set tidb_server_memory_limit 
SetGlobal: func(_ context.Context, s *SessionVars, val string) error {
	bt, str, err := parseMemoryLimit(s, val, val)
	if err != nil {
		return err
	}
	memory.ServerMemoryLimitOriginText.Store(str)
	memory.ServerMemoryLimit.Store(bt)
	gctuner.GlobalMemoryLimitTuner.UpdateMemoryLimit()
	return nil
}

To understand how the tidb_server_memory_limit variable affects TiDB’s MemoryTuner feature, let’s look at the (t *memoryLimitTuner) UpdateMemoryLimit function. t.GetPercentage() returns the value of tidb_server_memory_limit_gc_trigger.

// When `tidb_server_memory_limit` or `tidb_server_memory_limit_gc_trigger` changes, 
// t.UpdateMemoryLimit needs to be called to reset the runtime's MemoryLimit.
func (t *memoryLimitTuner) UpdateMemoryLimit() {
	var memoryLimit = t.calcMemoryLimit(t.GetPercentage())
	if memoryLimit == math.MaxInt64 {
		t.isTuning.Store(false)
		memoryLimit = initGOMemoryLimitValue  // -1
	} else {
		t.isTuning.Store(true)
	}
	debug.SetMemoryLimit(memoryLimit)
} 
| username: Fly-bird | Original post link

Since version v6.5.0, you can set the memory usage threshold for the tidb-server instance through the system variable tidb_server_memory_limit.

For example, to configure the total memory usage of the tidb-server instance to 32 GB:
SET GLOBAL tidb_server_memory_limit = “32GB”;

After setting this variable, when the memory usage of the tidb-server instance reaches 32 GB, TiDB will sequentially terminate the SQL operations with the highest memory usage until the memory usage of the tidb-server instance drops below 32 GB. The forcibly terminated SQL operations will return an error message Out Of Memory Quota! to the client.

| username: zhanggame1 | Original post link

The memory usage of tidb-server, you can use the top command and then press shift+m to see the tidb-server process.

| username: Soysauce520 | Original post link

Each TiDB server process has its own memory. In a distributed cluster, there isn’t a single parameter to configure the total memory. Each component has its own individual parameters. Generally, PD doesn’t need configuration, while KV has cache settings.

| username: tidb菜鸟一只 | Original post link

This is to separately set the memory for the tidb-server component, which occupies a percentage of the total memory of the server where the tidb-server is located. All memory settings are configured for individual components, such as tikv being set separately and tidb-server being set separately.

| username: TiDBer_小阿飞 | Original post link

This is a global-level parameter, right?
You mean a global parameter, but it only affects the TiDB module?

| username: Jolyne | Original post link

The memory usage of each server where tidb-server is located

| username: TiDBer_小阿飞 | Original post link

Got it.

| username: tidb菜鸟一只 | Original post link

Yes, this restricts the tidb-server component. There are also similar memory parameters that restrict tikv, and they apply to all tidb-server or tikv components.

| username: 随缘天空 | Original post link

The tidb_server_memory_limit parameter is used to set the memory limit for a single TiDB node, not the total memory usage of the entire TiDB cluster. In fact, there is no need to configure the total memory usage because it doesn’t make much sense. Each node’s configuration, memory, and CPU might be different. To maximize machine performance, you can set each node’s configuration individually based on the machine’s specifications. The specific parameters are as follows:

  1. Memory limit for TiDB nodes: By adjusting the tidb_server_memory_limit parameter for each TiDB node, you can control the memory usage of TiDB nodes.

  2. Memory limit for TiKV nodes: By adjusting the tikv_memory_limit parameter for each TiKV node, you can control the memory usage of TiKV nodes.

  3. Memory limit for PD nodes: By adjusting the memory limit parameters in the pd-server configuration file for each PD node, you can control the memory usage of PD nodes.

  4. Memory limit for TiFlash nodes: By adjusting the memory limit parameters in the tiflash-server configuration file for each TiFlash node, you can control the memory usage of TiFlash nodes.

When configuring these parameters, you need to balance the memory allocation based on actual conditions and hardware resources. Ensure that each component is allocated enough memory and avoid over-allocation that could lead to overall performance degradation.

Note that besides memory limits, other resource limits such as disk space and CPU should also be considered. By comprehensively considering these factors, you can better configure the total memory usage of the TiDB cluster.

| username: TiDBer_rYOSh9JN | Original post link

Is there a quicker way to adjust the total memory of a single TiDB instance (including TiDB, TiKV, PD, TiFlash)?

| username: 像风一样的男子 | Original post link

SET GLOBAL tidb_server_memory_limit = “32GB”; directly modify the system variable.

| username: Kongdom | Original post link

  1. This parameter should be for a single machine.
  2. Setting the total feels meaningless because a single machine might experience OOM.
    For mixed deployment, you can refer to this:
    三节点混合部署最佳实践 | PingCAP 文档中心
| username: gary | Original post link

Set the total memory usage for TiDB nodes.