Best Practices for Configuring a 3-Node Mixed Deployment in TiDB

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB 3节点混合部署配置最佳实践

| username: 超7成网友

3 nodes
96-core CPU, 256GB memory

Currently, each node is deployed with PD, TiDB, and TiKV.

How can the configuration be optimized, for example, how should the CPU be allocated?

| username: Kamner | Original post link

| username: 超7成网友 | Original post link

Thank you, I have read this document.
It mentions “80% of the CPU threads used by TiKV.”
What I am not sure about is, for example, if a node deploys TiDB, TiKV, and PD, and I have a node with 96 cores, how many CPUs should typically be allocated to TiKV?

| username: zhanggame1 | Original post link

You can consider NUMA binding for a 96-core CPU.

Based on my tests with version 7.5 for mixed deployment of 3 TiKV, 1 PD, and 1 TiDB, the key parameters are:

  • tikv.storage.block-cache.capacity
  • tidb.SET GLOBAL tidb_server_memory_limit = "32GB"

The appropriate values for these settings depend on how many TiKV and TiDB instances you plan to deploy.

| username: 超7成网友 | Original post link

Thank you, I have 3 nodes, each node deploys 1 TiDB, KV, and PD; do I still need to bind cores? I saw in the documentation that binding is required when multiple identical instances are deployed on one node.

Additionally: Are there any other default installations or parameters that need to be closed or adjusted for production?

| username: zhanggame1 | Original post link

You can test it. TiDB has TPCC and Sysbench stress testing components that can be deployed with TiUP. Then you can run it for a few days to see if it will OOM, and check the effect of binding or not binding the cores.

| username: tidb菜鸟一只 | Original post link

First, both TiKV and PD must have three instances each, and for availability considerations, they must be on separate servers. Since they all require persistence, I recommend using different disks to avoid I/O interference.

You only need to deploy 2 TiDB servers; even if one goes down, it won’t affect the business. Check if the machines have 2 or more NUMA nodes. If there are 2, I suggest deploying TiDB server and TiKV together and isolating them via NUMA. If there are 3, isolate TiDB server, PD, and TiKV via NUMA.

TiFlash can be omitted for now due to insufficient machines. Deploy other monitoring tools like Monitor and Grafana on the machine without the TiDB server to maximize resource utilization.

If NUMA nodes are insufficient or you want to limit parameters, set TiKV’s storage.block-cache.capacity to approximately 256/3*0.45≈40G to prevent OOM. For TiDB, set SET GLOBAL tidb_server_memory_limit = "80GB". If a node has PD, TiKV, and TiDB, set it to around 80G to avoid affecting other services.

| username: Kongdom | Original post link

In this situation, we generally deploy one PD, TiDB, and TiKV on each machine, and then use the default configuration.

| username: 超7成网友 | Original post link

There might be resource contention issues. I am considering limiting the memory usage for TiDB and TiKV. Is there any reference value for this, such as the ratio between TiDB and TiKV, or based on our 256GB memory, how much should we generally allocate to them according to experience?

| username: 超7成网友 | Original post link

Is it necessary to set the memory-usage-limit for TiKV? I saw in the community that TiKV’s memory is uncontrollable because this parameter is not set. What is the appropriate value to set for this?

| username: zhanggame1 | Original post link

Start with allocating less memory, leaving more available. Set storage.block-cache.capacity to 30G and tidb_server_memory_limit to 32G. Run it first and observe the memory usage of each process.

| username: tidb菜鸟一只 | Original post link

No need, just set storage.block-cache.capacity, and the total memory of TiKV will not exceed 2.5 times this value.

| username: RenlySir | Original post link

  1. Check if the server is NUMA, and if so, how many NUMA nodes there are.
  2. For mixed deployment, it is recommended to configure TiDB max-procs/memory to limit the CPU and memory usage of the TiDB server.
  3. Allocate 8G to 16G of memory for PD, and set the TiKV block-cache to 45% of the total memory minus the memory allocated for PD and TiDB.
  4. If the machine has high CPU and memory, it is recommended to consider deploying 2 TiKV + 1 TiDB or 2 TiDB + 1 TiKV on one machine, depending on the actual business needs.
| username: vcdog | Original post link

Deployment recommendations for a TiDB-cluster with 3 node servers are as follows:

  • TiDB-Server Deployment
  1. Node Allocation:

    • Deploy TiDB-Server on 2 of the nodes.
    • TiDB-Server can handle read and write requests and can automatically load balance between nodes.
  2. Resource Allocation:

Allocate sufficient CPU and memory resources on each node to run TiDB-Server. Given you have 96 cores and 256G memory, allocate resources based on actual workload needs. Typically, TiDB-Server does not require as many resources as TiKV but still needs enough to handle transactions and queries.

  • PD (Placement Driver) Deployment
  1. Node Allocation:

    • Deploy PD on all 3 nodes.
    • PD is responsible for maintaining the cluster state, including metadata and Placement Rules.
  2. Resource Allocation:

    • Since PD’s resource requirements are relatively low, allocate fewer CPU and memory resources on each node.
    • Allocate enough resources to ensure PD can efficiently handle cluster management tasks.
  • TiKV Deployment
  1. Node Allocation:

    • TiKV is a distributed storage system, and each node will store data.
    • Deploy TiKV on all 3 nodes.
  2. Resource Allocation:

    • TiKV is resource-intensive, requiring more CPU and memory resources on each node.
    • Considering you have 96 cores and 256G memory, allocate sufficient resources to each TiKV node to handle a large number of read and write operations.
  • Configuration Parameter Optimization

TiDB-Server:

  tidb:
    log.level: error
    new_collations_enabled_on_first_bootstrap: true
    performance.max-procs: 40
    performance.txn-total-size-limit: 42212254720 // If it's OLTP business, setting it too high may cause OOM
    prepared-plan-cache.enabled: true
    tidb_mem_oom_action: CANCEL
    tidb_mem_quota_query: 10737418240   // If it's OLTP business, setting it too high may cause OOM
    tmp-storage-path: /acdata/tidb-memory-cache
    tmp-storage-quota: -1

PD:
Adjust [cluster-version] in server.toml to match the versions of TiDB and TiKV.
Ensure [metric] configuration meets monitoring needs.

TiKV:

  tikv:
    coprocessor.region-max-size: 384MB
    coprocessor.region-split-size: 256MB
    raftstore.region-split-check-diff: 196MB
    raftstore.store-pool-size: 2
    readpool.coprocessor.use-unified-pool: true
    readpool.storage.use-unified-pool: false
    readpool.unified.max-thread-count: 48
    server.end-point-concurrency: 48
    server.grpc-concurrency: 48
    storage.reserve-space: 0MB
    storage.scheduler-worker-pool-size: 48
  • Improving Index Creation Efficiency
SET @@global.tidb_ddl_enable_fast_reorg=on;
SET @@global.tidb_ddl_reorg_batch_size = 10240;
SET @@global.tidb_ddl_reorg_worker_cnt = 32;
| username: Soysauce520 | Original post link

If you can use NUMA, bind the cores with NUMA for deployment. Assign one core per component, and it should be fine.

| username: TiDBer_ok0VXN1s | Original post link

Try this as the expert suggested: storage.block-cache.capacity

| username: zhanggame1 | Original post link

Isn’t tidb_ddl_reorg_worker_cnt = 32 too many?

| username: YuchongXU | Original post link

The workload is manageable.

| username: heiwandou | Original post link

Set CPU limits in the configuration.

| username: zhang_2023 | Original post link

Isn’t there a resource isolation feature?