TiDB Server Load 90%, TiKV Load 10%

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIDB Server 负载90%,TIKV负载10%

| username: TiDBer_Jzo3iMXn

[TiDB Usage Environment] Production environment
[TiDB Version] v6.5.1
[Reproduction Path] New environment set up a few days ago, old data was migrated from v3.1.0
[Encountered Problem: Problem Phenomenon and Impact]

Previously, there was an issue with analyze, causing the CPU load on a single node to be around 80% (not the tidb server node). Manually re-running analyze resolved this issue.

However, now the CPU load on the tidb server node often remains high, around 40% at best and up to 90% at worst;
The tidb server node only has tidb installed, without pd or other services;
The CPU load on the other 3 server nodes remains below 10%;
This issue did not occur with v3.1;
What could be causing this?

[Resource Configuration] There are 4 nodes

  1. 8 cores, 16 threads, 32GB, dedicated to tidb server
  2. 16 cores, 32 threads, 64GB, with tikv, pd, and tiflash deployed (although tiflash is installed, only a few tables use it)
  3. Same as 2
  4. Same as 2
| username: wakaka | Original post link

You can check what the connections are doing, there might be many slow query statements.

| username: lmdb | Original post link

Is there a system-level check? In a new environment, are there some resources running?

| username: TiDBer_Jzo3iMXn | Original post link

There are indeed some slow queries. What I don’t understand is why the TiKV node load is so low, while the TiDB node load is high :neutral_face:

| username: TiDBer_Jzo3iMXn | Original post link

The newly purchased server only has TiDB server installed, nothing else.

| username: tidb菜鸟一只 | Original post link

High load on TiDB is generally caused by slow SQL queries. It is recommended to optimize the SQL. If resources permit, adding two more machines to serve as TiDB servers and using a proxy for load balancing can also help alleviate the issue.

| username: TiDBer_Jzo3iMXn | Original post link

May I ask, is the data merging done in TiKV or TiDB?

| username: tidb菜鸟一只 | Original post link

TiDB, TiKV stores data for each region, and the final merging is done in TiDB.

| username: TiDBer_Jzo3iMXn | Original post link

In the end, it was found that there were some particularly long SQL statements, and the parsing of SQL is handled by the TiDB server, which caused its CPU load to be high. However, it is still unclear why this issue did not occur in version 3.1 before.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.