TIKV_REGION_STATUS had values for table_name yesterday, but today it is null

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIKV_REGION_STATUS昨天table_name有值，今天为null

| username: Hacker_kG69qSoN

[TiDB Usage Environment] Production Environment
[TiDB Version] 5.3.0
[Reproduction Path] Scheduled task truncates the table daily, then inserts over 40 million data entries.
[Encountered Problem: Symptoms and Impact]
Monitoring shows an increase in regions and memory, with a large number of empty regions generated at that time.
Through the TIKV_REGION_STATUS table, it is observed that regions with database and table names from yesterday have become null today. The data table was truncated and then new data was added.
This post TIKV_REGION_STATUS存在大量dbname table_name等字段都为null的记录 - TiDB 的问答社区 mentions it is a GC issue but does not specify how to resolve it.

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
region and memory growth.bmp (4.2 MB)
Empty regions merging quickly.bmp (2.1 MB)
TIKV_REGION_STATUS table_name had values yesterday, today it is null.bmp (4.5 MB)

2
| username: h5n1 | Original post link

show variables like ‘%gc%’; How many entries are there in mysq.delete_range/mysql.delete_range_done?

3
| username: Hacker_kG69qSoN | Original post link

I adjusted the time for gc_delete_range and gc_delete_range_done.
The data for gc_delete_range and done is from 1984.bmp (2.1 MB)

gc_delete_range: 9079
gc_delete_range_done: 146
gc configuration.bmp (2.1 MB)

4
| username: tidb菜鸟一只 | Original post link

If it hasn’t been garbage collected (GC), you need to wait for it to be GC’d first. After the GC is done, use tikvctl to check the size of the region corresponding to the region_id. Also, check the max-merge-region-size parameter to see how large it is set and whether it is small enough to require merging the region.

5
| username: tidb菜鸟一只 | Original post link

When converting, use tiup ctl:v5.4.3 pd -u https://10.10.10.16:2379 tso 445041280393412610
or directly SELECT TIDB_PARSE_TSO(445041280393412610);

6
| username: h5n1 | Original post link 
select * from mysql.tidb;

Run the pd-ctl service-gc-safepoint --pd pd_addr command to check the results.

The gc_delete_range is 9079 and the GC interval is 10 minutes, indicating that the GC is not executing properly.

7
| username: zhanggame1 | Original post link 
select * from mysql.tidb
Check the values of tikv_gc_run_interval, tikv_gc_last_run_time, and tikv_gc_safe_point.
8
| username: Hacker_kG69qSoN | Original post link

Currently, the GC appears to be normal.
gc execution status.bmp (2.4 MB)

9
| username: h5n1 | Original post link

Check the tidb.log on the leader tidb035 to see if there are many “gc worker busy” records related to GC.

10
| username: Hacker_kG69qSoN | Original post link

Indeed, there are many “gc worker is too busy” errors.

11
| username: Hacker_kG69qSoN | Original post link

I found the post you answered, and I am indeed on version 5.3.0. Can this only be resolved by upgrading? Is there a temporary manual solution?

12
| username: h5n1 | Original post link

The image is not visible. Please provide the text you need translated.

13
| username: Hacker_kG69qSoN | Original post link

Thank you, I found the reason. I also temporarily solved the problem by restarting TiKV. I will apply to see if we can upgrade to 6.5.5.

14
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.