Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: tikv节点大量Key is locked (will clean up) primary_lock
|
username: TiDB_C罗
[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version]
[Reproduction Path] What operations were performed to cause the issue
[Encountered Issue: Problem Phenomenon and Impact]
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]
Received TiKV_pending_task alert, checked the monitoring
Checked the TiKV node with the TiKV_pending_task alert and found the following log
[2024/06/27 02:11:35.641 +00:00] [WARN] [endpoint.rs:782] [error-response] [err="Key is locked (will clean up) primary_lock: 7480000000000001E15F72010E2B0E2D0E290E2FFF0E2B0E300E290E2BFF0E290E290E2F0E2BFF0E2E0E290E2D0E2CFF0E320E32021B0E2DFF0000000000000000F7 lock_version: 450744626864980002 key: 7480000000000001E15F698000000000000004040356E0FFF62FB5CF010E2B0E2D0E290E2FFF0E2B0E300E290E2BFF0E290E290E2F0E2BFF0E2E0E290E2D0E2CFF0E320E32021B0E2DFF0000000000000000F7 lock_ttl: 3000 txn_size: 1 lock_for_update_ts: 450744626864980002 use_async_commit: true min_commit_ts: 450744626864980003"]
Searched the forum and found related issues 专栏 - 一次 meet_lock 告警异常处理过程 | TiDB 社区, but I did not find any empty regions on PD
Using tiup ctl:v6.5.2 pd -u http://xxx.xxx.xxx.xxx:2379 -i
region key 7480000000000001E15F72010E2B0E2D0E290E2FFF0E2B0E300E290E2BFF0E290E290E2F0E2BFF0E2E0E290E2D0E2CFF0E320E32021B0E2DFF0000000000000000F7
{
"id": 118421,
"start_key": "7480000000000000FFF15F728000000000FFA68D4B0000000000FA",
"end_key": "7480000000000001FF1B5F728000000000FF0000060000000000FA",
"epoch": {
"conf_ver": 23,
"version": 691
},
"peers": [
{
"id": 118422,
"store_id": 248,
"role_name": "Voter"
},
{
"id": 118423,
"store_id": 249,
"role_name": "Voter"
},
{
"id": 118424,
"store_id": 250,
"role_name": "Voter"
}
],
"leader": {
"id": 118422,
"store_id": 248,
"role_name": "Voter"
},
"cpu_usage": 0,
"written_bytes": 135133,
"read_bytes": 957,
"written_keys": 60,
"read_keys": 0,
"approximate_size": 133,
"approximate_keys": 185869
}
Query TIKV_REGION_STATUS
mysql> SELECT * FROM INFORMATION_SCHEMA.`TIKV_REGION_STATUS` WHERE region_id=118421;
Empty set (0.71 sec)
This region does not exist, what could be causing this issue and how can it be resolved?