It seems that concurrency is insufficient

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 看起来并发不够

| username: TiDBer_uI8QIp1t

[TiDB Usage Environment] Production Environment
[TiDB Version] v7.1.2
[Reproduction Path] Changed the database connection of the business’s Spring Boot application to TiDB on server 222. When opening the page, there is a noticeable issue with interface processing.
[Encountered Problem: Phenomenon and Impact] When opening the page and checking with F12, the interface initially shows all as “append,” then changes to processed status in batches of 2-3. Switching back to MySQL noticeably speeds things up.
[Resource Configuration] Three servers: 8 cores, 32GB RAM, 512GB storage (IP: 183); 16 cores, 64GB RAM, 2TB storage (IP: 222); 16 cores, 64GB RAM, 2TB storage (IP: 201). Disks are XFS with write speeds up to tens of thousands of IOPS, all deployed on a gigabit internal network. Each node has db, pd, and kv deployed. Server 201, as the main controller, has monitoring services deployed. Currently in the trial phase, real-time incremental synchronization of MySQL data from server 222 to server 201 is set up using DM.
[Attachment: Screenshot/Log/Monitoring]

2
| username: zhanggame1 | Original post link

Write speed can reach 10,000 RPM? Mechanical disk? It’s better to use SSD for TiDB, mechanical disks are really not suitable.

3
| username: xfworld | Original post link

The network card is the bottleneck. You can check if the network on all nodes of the TiDB cluster is fully utilized.

Additionally, mixed deployment is quite challenging and difficult to achieve the desired balance without sufficient experience.

4
| username: TiDBer_uI8QIp1t | Original post link

I was wrong, it’s not ten thousand transfers. Using the sar command, the read/write can reach 10k per second.

5
| username: TiDBer_uI8QIp1t | Original post link

Moreover, it is very unstable. PD frequently crashes and KV often disconnects.

6
| username: 小龙虾爱大龙虾 | Original post link

What is the approximate concurrency, and what do the SQL queries generally look like?

7
| username: TiDBer_uI8QIp1t | Original post link

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CquhS6N223bGxSicDzwAAAYxO25quBM4AGBh8B84Er_GyCAAKBQvOAMCLXQ.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CquhS6N223bGxSicDzwAAAYxO25quBM4AGBh8B84Er_GyCAAKBQvOAMCLXQ.jpg_720x720q90720×540 48.2 KB

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8Cqj0h4WYsoIN66VQDzwAAAYxO28pHBM4AGBiiB84Er_GyCAAKBQvOATdm7A.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8Cqj0h4WYsoIN66VQDzwAAAYxO28pHBM4AGBiiB84Er_GyCAAKBQvOATdm7A.jpg_720x720q90720×540 57.1 KB

igEdAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqgxiHGy839Tc2X0DzwAAAYxO3CZ5BM4AGBjFB84Er_GyCAAKBQvOATdj_w.jpg_720x720q90
igEdAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqgxiHGy839Tc2X0DzwAAAYxO3CZ5BM4AGBjFB84Er_GyCAAKBQvOATdj_w.jpg_720x720q90720×540 67.5 KB

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqtV7zzBQHWyDQ_0DzwAAAYxO3JyYBM4AGBkIB84Er_GyCAAKBQvOASF24g.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqtV7zzBQHWyDQ_0DzwAAAYxO3JyYBM4AGBkIB84Er_GyCAAKBQvOASF24g.jpg_720x720q90720×540 61.5 KB

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqnFYFoV-godGceMDzwAAAYxO3SB5BM4AGBleB84Er_GyCAAKBQvOAU2QIA.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqnFYFoV-godGceMDzwAAAYxO3SB5BM4AGBleB84Er_GyCAAKBQvOAU2QIA.jpg_720x720q90720×540 43.1 KB

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqkTmpvTeTNvJv1QDzwAAAYxO3XgVBM4AGBmYB84Er_GyCAAKBQvOAUJOCA.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqkTmpvTeTNvJv1QDzwAAAYxO3XgVBM4AGBmYB84Er_GyCAAKBQvOAUJOCA.jpg_720x720q90720×540 43.4 KB

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8Cqv__9NSHIXZJq8QDzwAAAYxO3e7dBM4AGBnlB84Er_GyCAAKBQvOAJUtmg.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8Cqv__9NSHIXZJq8QDzwAAAYxO3e7dBM4AGBnlB84Er_GyCAAKBQvOAJUtmg.jpg_720x720q90720×540 57.5 KB

igELAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqifYXkqNn5eajR0DzwAAAYxO2yI3BM4AGBgwB84Er_GyCAAKBQvOAT1t3w.jpg_720x720q90
igELAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8CqifYXkqNn5eajR0DzwAAAYxO2yI3BM4AGBgwB84Er_GyCAAKBQvOAT1t3w.jpg_720x720q90720×540 62.3 KB

igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8Cqv__9NSHIXZJq8QDzwAAAYxO3e7dBM4AGBnlB84Er_GyCAAKBQvOAJUtmg.jpg_720x720q90
igEcAqNqcGcDAQTNEgAFzQ2ABtoAI4QBpCE82u8Cqv__9NSHIXZJq8QDzwAAAYxO3e7dBM4AGBnlB84Er_GyCAAKBQvOAJUtmg.jpg_720x720q90720×540 57.5 KB

8
| username: dba远航 | Original post link

Both the hard drive and the network card are bottlenecks.

9
| username: 春风十里 | Original post link

A lot of distributed resources are transmitted through network cards, and gigabit network cards might become a bottleneck.

10
| username: 小龙虾爱大龙虾 | Original post link

It seems that there aren’t too many SQL queries, and the concurrency isn’t very high, but the SQL isn’t running very fast. Let’s optimize those top ones; they are already at the second level.

11
| username: andone | Original post link

It is recommended to use a 10 Gigabit network card + SSD. Optimize the slow SQL.

12
| username: 像风一样的男子 | Original post link

The average execution time of your SQL is a bit alarming, prioritize optimizing the SQL.

13
| username: oceanzhang | Original post link

It feels like the resource usage at several points is also uneven.

14
| username: 有猫万事足 | Original post link

First, solve the issue of PD constantly crashing.
If PD keeps crashing, it is fundamentally unusable.

15
| username: TiDBer_uI8QIp1t | Original post link

SQL is indeed slow, but with such low concurrency, it feels like it’s not as fast as a standalone MySQL.

16
| username: TiDBer_uI8QIp1t | Original post link

The frequent crashes of PD might be due to KV disconnection.

17
| username: 有猫万事足 | Original post link

It is best not to place PD and TiKV on the same machine. If they are mixed, the memory parameters need to be adjusted.

If the stability issue is not resolved, testing is actually not very meaningful. No matter how high the performance, the system will crash, and these values will be meaningless. It is better to prioritize resolving the stability issue.

18
| username: 小龙虾爱大龙虾 | Original post link

It’s normal. TiDB is only compatible with MySQL, but that doesn’t mean if MySQL runs without issues, TiDB will too. They are fundamentally different databases, and different databases may require different approaches to achieve optimal performance.

19
| username: TiDBer_uI8QIp1t | Original post link

It won’t crash. The crash might have been due to using DM for data synchronization at the time, with a large volume of data being written, causing the crash.

20
| username: TiDBer_uI8QIp1t | Original post link

However, it’s not just that using 3 machines isn’t faster than running on a single machine, it’s actually slower than a single machine. This is not normal. Is there any configuration that can increase concurrency?