Which type of storage is more suitable for TiDB on K8S?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB on K8S 使用哪种的存储更合适？

| username: Kick

[TiDB Usage Environment] Production Environment / Testing / PoC

Production Environment

[TiDB Version]

  • v6.1.0

[Encountered Problem: Problem Phenomenon and Impact]

What is the recommended mounting method for external storage for TiDB on K8S?

[Resource Configuration]

Each node has approximately 1Ti x 10 SSD disks

[TiDB Operator Version]:

  • v1.3.4

[K8s Version]:

  • v1.20.11
2
| username: WalterWj | Original post link

Local NVMe manual dog head

3
| username: xfworld | Original post link

Isn’t Ceph Rook the standard configuration for K8S? But this is so complicated…

4
| username: Kick | Original post link

Is it directly using hostPath, or using local-volume-provisioner to create StorageClass, or something else?

5
| username: Kick | Original post link

Is your prod environment using Ceph Rook? Is the Ceph Rook cluster mixed with TiDB in the same k8s cluster? Or are TiDB and Ceph Rook in different k8s clusters?

6
| username: xfworld | Original post link

I don’t know how to use Ceph Rook myself, so I used hostPath directly.

Ceph is set up, but I haven’t tested it yet. :rofl:

7
| username: GreenGuan | Original post link

How can disk resources be limited when using hostPath?

8
| username: xfworld | Original post link

Not limited… :see_no_evil:

9
| username: Kick | Original post link

How can we ensure that the Pod can still be scheduled to the previous node after a restart when directly using hostPath? If it cannot be ensured, what is the cost of data synchronization, and has it been evaluated?

10
| username: xfworld | Original post link

NO, play by yourself, no evaluation has been done.

If you want to go into production, consider CEPH ROOK.

11
| username: GreenGuan | Original post link

Hmm, this seems to have entered a trade-off mode.
TiDB on k8s
hostPath: Solves the issue of automatic scaling on the operations side but cannot solve the resource isolation problem.
local-volume-provisioner: Solves the resource isolation problem but cannot solve automatic scaling (requires manual adjustment of replicas).
ceph: Solves both automatic scaling and resource isolation issues, but the problem is high operational costs and difficulty in troubleshooting storage component issues.

12
| username: tomsence | Original post link

The performance of local disks should be better.

13
| username: xfworld | Original post link

This can only be done by oneself, planned according to practice, and must be tested :smiley_cat:

There is no once-and-for-all solution.