TiDB Server Planning

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb服务器规划

| username: TiDBer_MWuedGIj

Hello everyone, I’m a newcomer. Recently, I’ve been looking into TiDB and reading the official documentation. I might have missed something, so I need the community’s help with a few questions.

Software version: tidb-7.1.1
Number of servers: 7
Components used: server, pd, tikv, grafana + monitoring
Server configuration: 7 identical machines, 256GB memory, 48 cores, 6*1.2T SSD disks
Disk planning: 1 disk for the system, 5 passthrough disks, no RAID
Architecture planning: 3 machines with 3 pd + 3 server, 1 machine for monitoring, 3 machines for tikv

Question 1: How can I use this server configuration to ensure maximum utilization?
Question 2: I read in community posts that tikv only supports SSDs below 2T mounted to a specific directory, but with multiple tikv disks, it feels wasteful. However, with 3 tikv machines, there can be multiple tikv instances. If one tikv machine shuts down, won’t the replicas only exist on the three instances of that single machine? If that’s the case, wouldn’t it cause data loss?
Question 3: Can the disks on the pd and server machines be configured as RAID5 for larger disk space? Or can they also be used as tikv instances? It feels like the machines are being wasted.

| username: xfworld | Original post link

TiUP requires a node, and the configuration can be a bit lower.

| username: dba-kit | Original post link

  1. The models are too limited, all of them are storage types. With TiPD + TiDB, there’s no need for large disks, 100G SSD is more than enough, the remaining SSDs are just wasted.
  2. TiKV does not have a restriction to be below 2T, you can try using Raid 0 or striped LVM to see which one performs better. Similarly, the monitoring node can be shared with the TiUP node, as suggested by xfworld.
| username: xfworld | Original post link

Question 1:
Disks can be used directly without RAID (don’t consider hybrid deployment, optimization will be more difficult). How to use them depends on your scenario…
If it were me, I might consider virtualization and directly use a virtual environment. Handling it through VM can save some resources.

Question 2:
Regarding the capacity issue of TiKV, you need to consider the business scenario. If there is a massive amount of data, you need to consider hot and cold data separation. This way, SSD resources can be utilized.
The number of TiKV instances and replicas is crucial for ensuring high availability. When the number of instances is sufficient but the number of replicas is insufficient, PD will automatically schedule to make up the number of replicas. Therefore, you should prioritize whether the number of instances is sufficient.

Question 3:
There is no need for large disk space for PD and TiDB, as it is not conducive to the rapid recovery solution of multiple instances, and the high availability capability is also reduced.
Basically, if RAID5 fails, the cluster will be down… :melting_face: :melting_face: :melting_face: :melting_face:

| username: dba-kit | Original post link

This is actually possible, but if divided into 6 TiKV instances and isolated through Cgroup, each TiKV’s resources will be smaller, making it easy to waste resources when hotspots occur. (Or if not isolated, it is prone to OOM). As for the concern about multiple replica loss, you can force multiple replicas of a region to be distributed to different instances (hosts) through replication.location-labels: ["host"].

You can refer to the official documentation: 混合部署拓扑 | PingCAP 归档文档站

| username: dba-kit | Original post link

If allowed, it is recommended to replace three machines with six machines, with three higher-performance machines used as TiDB Servers and three lower-performance machines used as TiPD (8C16G can support at least a 100T cluster). A 100G SSD data disk is sufficient.

| username: Fly-bird | Original post link

With the configuration unable to change, 2 TiDB, 3 PD, and 3 KV, where TiDB and PD are placed together, TiUP + monitoring are placed together with PD, and KV is placed separately.

| username: 像风一样的男子 | Original post link

Architecture Planning: 3 machines, 3 PD + 3 server, 1 monitoring machine, 3 TiKV. This setup seems a bit wasteful. PD and TiDB do not have such high disk performance requirements. The first three machines can be set up with RAID and then virtualized. Use virtual machines to install PD, TiDB, and monitoring. The KV servers should not be virtualized. Each server should have 4 SSDs, with each disk installing one KV instance, allowing each server to install 4 KV instances.

| username: tidb菜鸟一只 | Original post link

Number of servers: 7
Components used: server, pd, tikv, grafana + monitoring
Server configuration: 7 identical servers, 256G memory, 48 cores, 6*1.2T SSD disks
Disk planning: 1 disk for the system, 5 passthrough disks, no RAID
Architecture planning: 3 machines, 3 pd + 3 server, 1 monitoring, 3 tikv

Question 1: How to use this server configuration to ensure maximum utilization?
If the number of machines is fixed at 7, you can use 2 machines for pd + tidb-server, 1 machine for pd + monitoring + tiup, and the remaining 4 machines for tikv. It is recommended to use LVM to merge the disks into directories.

Question 2: I read in the community posts that tikv only supports SSDs below 2T mounted to a certain directory, but TIKV has multiple disks, which feels wasteful. However, with 3 tikv machines, you can have multiple tikv instances. If one tikv machine shuts down, will the replicas not only exist on the three instances of that one machine? If so, wouldn’t that cause data loss?
You can set up 2 instances on each tikv machine, using NUMA resource isolation. Each machine can use LVM to mount 2 directories, one for each instance, and then use labels to specify the host storage. This way, multiple replicas of a region will not be stored on the same host.

Question 3: Can the disks on the pd and server machines be configured as RAID5 for large disk space mounting? Or can they also be used as tikv instances? It feels like the machines are being wasted.
TiDB is a distributed database, and tidb-server is stateless and does not need to store data. Since pd has three nodes, it doesn’t matter if one is down, so there’s no need to do RAID5. I suggest using LVM directly.

| username: TiDBer_MWuedGIj | Original post link

Thank you for your reply.
My servers are all hardware-based, ARM architecture, and currently not virtualized.
I carefully reviewed the hybrid deployment last night.
I saw a reply below saying that the disk can be configured as LVM, because a single SSD disk is only 1.2T, and I saw the official documentation mentioning:
• TiKV disk size configuration recommendation: PCI-E SSD should not exceed 2 TB, regular SSD should not exceed 1.5 TB.
Question: Will there be any impact if the disk size exceeds this limit?
Question: If I have 3 TiKV instances, since the machine configuration is relatively high, can I remove a few hard disks, leaving 3 separate hard disks on each machine, and then start 3 TiKV instances on each machine to use different fast hard disks as data disks? For example, each machine’s three disks correspond to sdb, sdc, sdd
/dev/sdb /data1
/dev/sdc /data2
/dev/sdd /data3

| username: TiDBer_MWuedGIj | Original post link

Thank you for your reply. The machine is a physical machine, and virtualization is not currently being used.
I saw in the official documentation that it is recommended:
• TiKV disk size configuration suggests PCI-E SSD not exceeding 2 TB, and regular SSD not exceeding 1.5 TB.
If what you say is correct, my resources are indeed too wasted.
The configuration is very high; can I run 3 TiKVs on one machine? Then use labels for region correction.

| username: TiDBer_MWuedGIj | Original post link

Yes, 3 TiKV nodes, each with 4 separate disks, each starting 4 TiKV instances, with each TiKV using one data disk as the data directory, right?
Then use labels for registration isolation to prevent 2 replicas from running on one machine.

| username: tidb菜鸟一只 | Original post link

This configuration can support 3 TiKV instances. It is recommended that each TiKV instance has at least 16 cores and 64GB of memory. Be sure to use labels for region isolation; otherwise, if all three replicas of a region are stored on one machine and that machine fails, you will lose data. In practice, the resources for PD and TiDB servers are somewhat wasted, but it is recommended not to place TiKV together with them.

| username: TiDBer_MWuedGIj | Original post link

Well, does TiKV have instance limitations on a single machine?

| username: 像风一样的男子 | Original post link

The person above is right. It is recommended to have at least 16 cores and 64GB of memory for a TiKV instance. So you can run 3 KV instances on one physical machine, as long as each KV instance has a different startup port on the same machine.

| username: TiDBer_MWuedGIj | Original post link

Understood. Then just configure the label and bind the CPU.

| username: TiDBer_MWuedGIj | Original post link

I have also seen PD and server running on Kubernetes, while TiKV runs separately on physical machines.

| username: 像风一样的男子 | Original post link

Memory usage limit for each KV

| username: TiDBer_MWuedGIj | Original post link

Okay, thank you.

| username: TiDBer_MWuedGIj | Original post link

Hello, the recommended disk size configuration for TiKV is not to exceed 2 TB for PCI-E SSDs and 1.5 TB for regular SSDs. Can this limit actually be exceeded?