Is there any recommended deployment plan for a new cluster with 30TB of data, and how many TiKV nodes are needed?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 新部署一个有30T数据的集群,这块有没有啥推荐的部署方案,tikv节点需要多少呢

| username: terry0219

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 7.5.1
Deploying a new cluster with 30TB of data, are there any recommended deployment plans for this? How many TiKV nodes are needed, and how much storage is required for each TiKV? Please provide some guidance.

| username: Jellybean | Original post link

If machine resources are sufficient, you can refer to the official recommended deployment plan:

If single machine resources are average, deploy one node per machine.

| username: 像风一样的男子 | Original post link

For such a large cluster going online, have you considered purchasing a commercial version as a fallback?

| username: 啦啦啦啦啦 | Original post link

We have a cluster with a data volume of a little over 30TB, with a total of 32 KVs. The disk usage rate for a single KV is about 50%, for reference.

| username: DBAER | Original post link

Considering the future data growth and the corresponding scenario, whether it is AP or TP, it is generally recommended to follow the official suggestions for independent deployment, and you can also conduct business stress testing.

| username: terry0219 | Original post link

How large is the disk for a single TiKV?

| username: terry0219 | Original post link

Mainly due to limited budget :joy:

| username: 啦啦啦啦啦 | Original post link

Our setup uses 1.5 TB NVMe SSDs for each individual disk.

| username: 像风一样的男子 | Original post link

Are you using a cloud server or an on-premises data center?

| username: terry0219 | Original post link

What version is your cluster? With so many TiKV nodes, how many replicas are you using?

| username: terry0219 | Original post link

Cloud server

| username: 啦啦啦啦啦 | Original post link

3 replicas of version 6.1

| username: 啦啦啦啦啦 | Original post link

Cloud servers are easy to manage; you can just scale as needed. Mine is a physical machine, and procurement is very troublesome, so the disk space has to be very redundant to ensure growth over the next few years.

| username: Jellybean | Original post link

To deploy a cluster, you can check what machines are available. If you want fewer machines, then each disk needs to be larger, and the memory and CPU configuration of a single machine should be higher. In this case, you can deploy 2 to 3 nodes of the same type on a single machine.

| username: terry0219 | Original post link

Yes, it’s more convenient to expand cloud servers.

| username: terry0219 | Original post link

I want to use a 3.5T NVMe SSD, so the number of machines can be reduced.

| username: Jellybean | Original post link

Roughly calculated, the estimated number of TiKV machines needed is 9 = 30TB data / 50% disk usage / 3.5TB per disk / 2 disks per machine.
If you want a lower disk usage rate, then the number of machines should be appropriately increased.
If the memory and CPU of a single machine are high enough, such as 256GB memory and 100vCPU, you can evaluate attaching 3 disks per machine based on the business situation.

Additionally, add 3 PD nodes. Evaluate whether TiDB nodes can be considered for mixed deployment with PD based on the business situation. If the business has heavy AP analysis or OOM risk, then deploy TiDB independently.

Monitoring and control nodes can be deployed on 1 machine, placing them together. TiFlash and TiCDC can be determined based on business needs.

| username: zhanggame1 | Original post link

A single machine can run multiple TiKVs, with one TiKV per hard drive. As long as the resources are sufficient and the parameters are properly configured, there should be no problem.

| username: 友利奈绪 | Original post link

Just make sure the resource usage doesn’t explode, and test it first.

| username: 我是人间不清醒 | Original post link

Could you please tell me how many of your 32 nodes are allocated to TiKV, TiDB server, PD, and TiFlash respectively?