Only One Node's Disk IO is Fully Utilized

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 节点只有一个磁盘io打满

| username: Hacker_6ASfgBFe

Question: Only one node’s disk IO is fully utilized.

| username: Hacker_6ASfgBFe | Original post link

At this time, I checked the slow log and found a delete operation on the partitioned table, with the slowest execution taking more than 100 seconds.

| username: zhanggame1 | Original post link

Take a look at the traffic visualization on the dashboard to see which region has hotspots during this period.

| username: Hacker_6ASfgBFe | Original post link

| username: zhanggame1 | Original post link

The traffic visualization chart on the dashboard

| username: Hacker_6ASfgBFe | Original post link


Can’t view it

| username: zhanggame1 | Original post link

Don’t choose 1 day.
Use the box selection feature to select the time point you are interested in.

| username: Hacker_6ASfgBFe | Original post link

How to locate the point in time from yesterday?

| username: Hacker_6ASfgBFe | Original post link

Is it possible to check through other monitoring information? This thing is a bit complicated and hard to understand.

| username: Jellybean | Original post link

It should be a hotspot issue. Search in the forum for how to handle hotspot issues, and there will be solutions.

| username: gcworkerishungry | Original post link

  1. Optimization of the delete operation itself, such as indexing and deletion methods; 删除数据 | PingCAP 文档中心
  2. Optimization of delete time: whether it can be adjusted;
  3. Strategy adjustment: TTL feature
| username: changpeng75 | Original post link

What is the version of TiDB?
How do you configure it to scatter regions to avoid hotspot issues?

| username: ShawnYan | Original post link

How many rows of data are deleted in a transaction?

| username: WinterLiu | Original post link

Issues such as data hotspots

| username: 随便改个用户名 | Original post link

How about trying to use non-transactional deletion?

| username: 路在何chu | Original post link

Check if the hotspot regions of this table are concentrated on this table.

| username: xingzhenxiang | Original post link

It looks like there are quite a few keys. Are there too few TiKV nodes? Have you deleted the data from the compression layer?

| username: 哈喽沃德 | Original post link

iostat -x

| username: 双开门变频冰箱 | Original post link

Could it be hot data?

| username: Kongdom | Original post link

Is the entire cluster on a single node? Is it using a mechanical hard drive? If you are deleting a large amount of data and it is a mechanical hard drive, it should be normal.