Confirming Write Hotspot Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 写入热点问题确认

| username: TiDBer_x2jH2Nr5

[TiDB Usage Environment] Production Environment
[TiDB Version] v4.0.8
[Encountered Problem: Symptoms and Impact]
During heavy writes, according to TiDB 热点问题处理 | PingCAP 文档中心, the Raftstore CPU appears normal (screenshot below), and the Store write rate bytes also seem normal (screenshot below). However, the Region written related panel seems to indicate a potential issue (screenshot below). I searched the forum but couldn’t find information on how to interpret the Region written panel. I want to confirm if there is a hotspot issue.

Currently, the primary key uses AUTO_INCREMENT.

[Resource Configuration]

[Attachments: Screenshots/Logs/Monitoring]
Raftstore CPU:

Store write rate bytes:

Region written related:

| username: xfworld | Original post link

You can easily see it through the traffic visualization on the dashboard.

| username: TiDBer_x2jH2Nr5 | Original post link

According to the visualization, it seems that there is no situation matching TiDB 热点问题处理 | PingCAP 文档中心, there is no such step-like pattern. Is this the correct interpretation?

| username: 有猫万事足 | Original post link

The write hotspot map does not show bright diagonal lines, and the read hotspot map does not show bright horizontal lines.

If this is your write hotspot map, there should be no hotspot issues.

It is possible that your storage throughput capability is quite strong, so you can try lowering the brightness a bit and see. If it still remains this bright overall, it seems like the overall resources are tight, and you might need to add more TiKV nodes.

Especially the stat…grams table, its distribution on the x-axis is always very bright, which is an enhanced version of the alternating brightness described in the documentation. :joy:

| username: TiDBer_x2jH2Nr5 | Original post link

Setting the brightness to -1 looks like this:

And the TiKV has 8 CPU cores and 32GB of memory.
The situation of TiKV at that time was like this:

Is this situation a sign of resource strain?

| username: 有猫万事足 | Original post link

From this graph, it looks like your reads are much greater than your writes, by a factor of 40. It doesn’t seem like there’s a bottleneck in writing. It seems more like some process is crazily extracting data from your TiDB.

| username: xfworld | Original post link

This indicates that all the data is being utilized, which is the best state.

| username: TiDBer_x2jH2Nr5 | Original post link

Yes, the business is like this, but now during high concurrent inserts, the maximum QPS for insert on each TiDB is only around 6k. So I want to check if there are any hotspot issues or something.

| username: 有猫万事足 | Original post link

Look at my poor configuration. When importing a large table, this is the data. Although overall, the maximum read value is indeed higher than the write value, if it were a write-intensive situation as you suspect, the data in this graph shouldn’t look like this.

| username: hey-hoho | Original post link

Looking at the pulse, there is no write hotspot.

| username: 有猫万事足 | Original post link

I checked the 4.0 documentation

It seems that this version does not have the setting for non-clustered tables. In scenarios where you use AUTO_INCREMENT, there is no way to set it to non-clustered table + SHARD_ROW_ID_BITS.
The documentation suggests directly replacing AUTO_INCREMENT with AUTO_RANDOM.

| username: TiDBer_x2jH2Nr5 | Original post link

Yes, previously I planned to replace AUTO_INCREMENT with AUTO_RANDOM if it was confirmed to be a hotspot issue. Since the table is quite large and there are many of them, I wanted to first confirm whether there is a hotspot issue, which is why I posted this. If there is no hotspot issue, is it unnecessary to replace it?

| username: xfworld | Original post link

No hot topics, why bother… :rofl:

| username: 有猫万事足 | Original post link

First of all, the graph you provided for this time period definitely does not indicate a write hotspot. However, it’s hard to say whether high write concurrency has caused a hotspot. After all, using AUTO_INCREMENT is not recommended in the documentation.

If your old business has already used it and has a lot of data running on it, don’t change it unless absolutely necessary. You can closely monitor it. This is a potential risk. As long as nothing goes wrong, you can treat it as if nothing happened. :joy:

For newly created tables, follow the documentation recommendations if possible. Especially when you are testing, you can fully test how much this change improves the insert performance. Practice brings true knowledge.

| username: TiDBer_x2jH2Nr5 | Original post link

Sure, thanks. Because I didn’t do Sysbench before, I don’t know what the maximum QPS for insert is now. :joy:

| username: TiDBer_x2jH2Nr5 | Original post link

Is the readWrite mentioned in TiDB Sysbench 性能对比测试报告 - v5.0 对比 v4.0 | PingCAP 文档中心 referring to read-write concurrency?

| username: xfworld | Original post link

Of course, you can probably look at the picture.


The place marked with an arrow is the number of threads, which is the concurrency…

| username: TiDBer_x2jH2Nr5 | Original post link

Okay, thank you.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.