Meaning and Calculation Methods of Some Fields in the TiDB System View TIDB_HOT_REGIONS

translator_bot · June 21, 2024, 5:30pm

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIDB 系统视图TIDB_HOT_REGIONS中部分字段含义&计算方式

| username: residentevil

【 TiDB Usage Environment】Production Environment
【 TiDB Version】v6.1.7
【Encountered Problem: Problem Description and Impact】What are the specific meanings of the MAX_HOT_DEGREE and FLOW_BYTES fields in the TIDB_HOT_REGIONS table in the information_schema database? I hope to obtain the regions with hotspots from this table and then perform the corresponding migration.

translator_bot · June 21, 2024, 5:30pm

| username: 昵称想不起来了 | Original post link

github.com

pingcap/docs/blob/master/information-schema/information-schema-tidb-hot-regions.md?plain=1

---
title: TIDB_HOT_REGIONS
summary: Learn the `TIDB_HOT_REGIONS` information_schema table.
---

# TIDB_HOT_REGIONS

The `TIDB_HOT_REGIONS` table provides information about the current hot Regions. For information about history hot Regions, see `[TIDB_HOT_REGIONS_HISTORY](/information-schema/information-schema-tidb-hot-regions-history.md)`.

> **Note:**
>
> This table is only applicable to TiDB Self-Hosted and not available on [TiDB Cloud](https://docs.pingcap.com/tidbcloud/).

{{< copyable "sql" >}}

```sql
USE information_schema;
DESC tidb_hot_regions;
```

This file has been truncated. show original

MAX_HOT_DEGREE: The maximum hot degree of the Region.
FLOW_BYTES: The number of bytes written and read in the Region.

translator_bot · June 21, 2024, 5:30pm

| username: residentevil | Original post link

The explanation is quite clear. It seems that we can identify hotspot REGIONS using this method. If we know which REGION it is, can we manually trigger the REGION to split?

translator_bot · June 21, 2024, 5:30pm

| username: 有猫万事足 | Original post link

The algorithm for the hot degree is quite interesting.

First, regions are logically divided into buckets.

github.com

tikv/rfcs/blob/master/text/0082-dynamic-size-region.md#bucket

# Dynamic size region

- RFC PR: https://github.com/tikv/rfcs/pull/0082
- Tracking Issue: https://github.com/tikv/tikv/issues/11515

## Summary

Make the size of a region dynamic, and only set a upper limit.

This is the first step that we try to support PiB scale cluster.

## Motivation

We have observed a lot of regressions when the count of regions increases. The major drawback
comes from three aspects:

1. Transactions need to access more regions, which lead to too many RPCs and have high latency;
2. A node needs to drive more regions, which can have high resource usages and hurt performance;
3. Tools and service like CDC/BR/GC need to loop over all regions, which is slow.

This file has been truncated. show original

Then TiKV reports six metrics for each bucket: read bytes, read keys, read QPS, written bytes, written keys, and written QPS.

After reporting to PD, these metrics are compared with a static baseline. The specific settings for the static baseline can be found at

github.com

tikv/pd/blob/master/pkg/statistics/utils/constant.go#L39-L46


      
          var MinHotThresholds = [RegionStatCount]float64{
          	RegionReadBytes:     8 * units.KiB,
          	RegionReadKeys:      128,
          	RegionReadQueryNum:  128,
          	RegionWriteBytes:    1 * units.KiB,
          	RegionWriteKeys:     32,
          	RegionWriteQueryNum: 32,
          }

The algorithm is located at

github.com

tikv/pd/blob/1899f41b8842acee888ec618e4a8dcceb1f231ca/pkg/statistics/buckets/bucket_stat_informer.go#L188


      
          		}
          		// move the left item to the next, old should move first if they are equal.
          		if bytes.Compare(newItem.EndKey, oldItem.EndKey) > 0 || len(newItem.EndKey) == 0 {
          			p2++
          		} else {
          			p1++
          		}
          	}
          }
          
          func (b *BucketTreeItem) calculateHotDegree() {
          	for _, stat := range b.stats {
          		// TODO: qps should be considered, tikv will report this in next sprint
          		// the order: read [bytes keys qps] and write[bytes keys qps]
          		readLoads := stat.Loads[:2]
          		// keep same with the hot region hot degree
          		// https://github.com/tikv/pd/blob/6f6f545a6716840f7e2c7f4d8ed9b49f613a5cd8/pkg/statistics/hot_peer_cache.go#L220-L222
          		readHot := slice.AnyOf(readLoads, func(i int) bool {
          			return readLoads[i] > minHotThresholds[i]
          		})
          		writeLoads := stat.Loads[3:5]

As you can see, if any of the three read/write metrics exceed the baseline value, the hot degree of the current bucket is incremented by 1; similarly, if none of the metrics exceed the baseline value, the hot degree is decremented by 1. Additionally, the hot degree of a single bucket has an upper and lower limit, ranging from [20, -20], and it will not fluctuate beyond this range. The hot degree of a region is the sum of the hot degrees of the logical buckets.

My understanding is limited, but this is roughly the idea.

translator_bot · June 21, 2024, 5:30pm

| username: Fly-bird | Original post link

READ_BYTES: The estimated amount of data read in a Region within one heartbeat cycle, measured in bytes. From this, it can be seen that Regions with particularly large values are experiencing hotspots.

translator_bot · June 21, 2024, 5:30pm

| username: residentevil | Original post link

The analysis is very thorough. Next, I’ll look into how to balance the hotspot regions.

translator_bot · June 21, 2024, 5:30pm

| username: 大飞哥online | Original post link

Got it