Why is the memory distribution uneven among the three deployed hosts?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 为什么部署的三台主机的内存分布不均匀?

| username: SummerGu

[TiDB Usage Environment] Production Environment
[TiDB Version] 5.4.3
In our production deployment cluster, the memory usage seems to be a bit problematic. One machine is using very little memory. What could be the issue or how can it be resolved?


| username: 像风一样的男子 | Original post link

Why is your CPU fluctuating like a roller coaster? Did TiDB perform load balancing?

| username: Kongdom | Original post link

In a three-node mixed deployment, the node where the PD Leader is located should relatively use more resources.

| username: 心在飞翔 | Original post link

You can see that your server 38 has recently restarted, and the cache of TiKV has been cleared, so its memory usage is smaller than the other two TiKVs. After running for a while, it will basically level out.

| username: 小龙虾爱大龙虾 | Original post link

The block cache in TiKV, once used, will not be returned to the operating system unless it exceeds the limit. Other experts have also noticed this. After 38 restarts, combined with potential load imbalances, such as uneven connection numbers on TiDB nodes, the PD itself has a high leader load.

| username: SummerGu | Original post link

The roller coaster CPU is a TiFlash node :joy:

| username: Billmay表妹 | Original post link

TiDB and PD can be placed together.

TiKV should be deployed separately, one node per machine, without mixing.

Check the resource status of the host list~

| username: 像风一样的男子 | Original post link

Mixed deployment of resources cannot be uniform, leading to many issues.

| username: xiexin | Original post link

When the CPU spikes, check which process is consuming the most resources on the machine. It looks like some large queries.

| username: tidb菜鸟一只 | Original post link

I suspect that your business load has been concentrated on a particular TiDB, causing the memory usage on that specific machine to be particularly high.

| username: 有猫万事足 | Original post link

Go to the topsql page and check if the executed SQLs are similar? I feel there should be significant differences, as the business might not be evenly distributed. From the load balancing software, are the connection numbers balanced?

| username: zhanggame1 | Original post link

In a mixed deployment, TiKV memory will gradually increase until it reaches the limit. TiDB may experience a sudden significant increase due to large queries.

| username: andone | Original post link

It is probably caused by mixed deployment. Mixed deployment is not recommended.

| username: WalterWj | Original post link

If deployed in a hybrid manner, have you implemented resource control for TiDB and TiKV?

| username: 春风十里 | Original post link

Use the top command on each node and then press shift+m to check the memory usage order. Compare the three nodes, and you should be able to identify which functional component is causing the memory difference.

| username: dba远航 | Original post link

It should be caused by the uneven distribution of REGION leaders.