Issues with Physical Machine Disks in TiDB-TiKV

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 关于TIDB-TIKV物理机磁盘问题

| username: TiDBer_MWuedGIj

[TiDB Usage Environment] Production Environment / Testing / PoC
Production Environment
[TiDB Version]
Latest Version
[Reproduction Path] What operations were performed to encounter the issue
[Encountered Issue: Issue Phenomenon and Impact]
Hello everyone, I have a question and seek help from those with experience. According to the official documentation, it is recommended that the size of TiKV disks should not exceed 2 TB for PCI-E SSDs and 1.5 TB for regular SSDs.

The servers we are planning to purchase have single SATA interface SSDs. If the size is less than 1.5 TB, the only option is a 960 GB disk, which is too small in capacity.
If I use a 1.92 TB SATA interface SSD, what impact will it have?
Usage Plan: Each machine will run 3 TiKV instances and mount multiple SSD disks.
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]

| username: zhanggame1 | Original post link

It doesn’t have much impact. This is about the recommended data volume for TiKV, not the large sales volume of physical hard drives.

| username: 大飞哥online | Original post link

When TiKV processes large queries or generates data structures in memory, it will occupy a portion of the system memory. If the SSD size is too large, it may lead to increased disk I/O operations and longer system response times, thereby affecting the performance of the entire TiDB cluster.

| username: 大飞哥online | Original post link

How much does it specifically affect? I haven’t tested it. To put it bluntly, a single server shouldn’t have too many or too large disks attached, as it also requires system memory.

| username: TiDBer_MWuedGIj | Original post link

My hard drive is large, so the data volume is definitely large. I have 3 nodes, each with a 1.92TB SSD. Over the years, the data volume has naturally become very large.

| username: TiDBer_MWuedGIj | Original post link

Can I use a SATA drive with a single disk of 1.92T? My server has high memory and CPU, with 246G and 48 cores. I plan to use 3 SSDs and set up 3 instances.

| username: Fly-bird | Original post link

You can choose not to accept the official recommendations.

| username: 大飞哥online | Original post link

Forget about SATA; for databases, disk I/O is the most critical factor. It’s best to use a better one.

| username: TiDBer_vfJBUcxl | Original post link

The official recommendation is that it can be not adopted.

| username: Jellybean | Original post link

The official recommendation is to use NVMe type SSDs in production environments. While using single SATA interface SSDs is not prohibited, their performance will not be optimal. The key issue is not the smaller capacity.

For development environments, simple functionality verification, or when resources are really limited, SATA drives can be used, but performance will be significantly reduced, and the effectiveness of TiDB may not meet expectations.

In short, decide your configuration resources based on your application scenario and cost objectives.

| username: zhanggame1 | Original post link

I strongly recommend NVMe SSDs, the performance difference is too significant. For servers, buy U.2 interface ones.

| username: TiDBer_MWuedGIj | Original post link

Thank you. May I ask, what size of SSD is suitable for NVM? Not considering the cost for now.

| username: cassblanca | Original post link

No budget constraints, just go for Intel Optane. 4K random read/write is unbeatable. :grinning:

| username: Kongdom | Original post link

This is an official recommendation, not an official requirement.
But running 3 TiKV instances on one server, aren’t you worried about resource contention?

| username: zxgaa | Original post link

The official guidelines are just best practice recommendations. We have a 4TB SSD, and it’s working perfectly fine.

| username: zxgaa | Original post link

Resource isolation can be done.

| username: Kongdom | Original post link

Well, although resource isolation is possible, it’s not as straightforward as physical isolation. :yum:

| username: TiDBer_小阿飞 | Original post link

It is recommended to use NVME interface SSDs for TiKV to ensure faster read and write speeds.

PCIe 5.0 has not been released yet, and PCIe 4.0 is relatively more expensive, so the mainstream choice is still PCIe 3.0 NVME protocol SSDs, which offer a higher cost-performance ratio.