Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: tikv 一节点磁盘损坏，无法在使用，如何完整下线这个tikv 节点？
[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed that led to the issue
[Encountered Issue: Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots / Logs / Monitoring]
Last night, a disk failure occurred on one of the TiKV nodes (a total of 4 TiKV nodes) in an online TiDB cluster version V5.4.0. The disk could not be mounted and was unusable. It was only discovered this afternoon that this TiKV node was in a down state. Preliminary investigation: Business operations are normal, and the disk usage of the other 3 TiKV nodes has increased (presumably because the data has been rebalanced). First, I executed stop on the damaged TiKV node, which succeeded normally. Then I performed scale-in, which prompted pending offline. After consulting the documentation, I executed scale-in --force. The damaged node was no longer visible in the display, but it still showed up in pd store with the status offline. I executed pd store delete xxx (the TiKV node ID), but it still exists. How can I remove this damaged TiKV node from pd?
After executing force, does the PD store show “physical destroyed = true”? If so, then there’s no problem. Just create a new one directly.
pdctl> store remove-tombstone
Give it a try.
Currently, this damaged TiKV is displayed in PD as follows, and I did not see physical destroyed = true:
{
"store": {
"id": 226195121,
"address": "xxxx:20160",
"state": 1,
"version": "5.4.0",
"status_address": "xxxx:20180",
"git_hash": "b5262299604df88711d9ed4b84d43e9c507749a2",
"start_timestamp": 1673589464,
"deploy_path": "/data1/tidb-deploy/tikv-20160/bin",
"last_heartbeat": 1680086231611976151,
"state_name": "Offline"
},
"status": {
"capacity": "2.865TiB",
"available": "1.149TiB",
"used_size": "1.367TiB",
"leader_count": 0,
"leader_weight": 1,
"leader_score": 0,
"leader_size": 0,
"region_count": 43366,
"region_weight": 1,
"region_score": 3227273,
"region_size": 3227273,
"slow_score": 1,
"start_ts": "2023-01-13T00:57:44-05:00",
"last_heartbeat_ts": "2023-03-29T05:37:11.611976151-05:00",
"uptime": "1804h39m27.611976151s"
}
}
This is specifically for the status of tombstone. As you can see above, it is currently still in offline status.
Here it hasn’t reset to zero. Wait until there are no abnormal regions in the cluster and check again. It is possible that the region3 replicas on other nodes have not been fully restored yet.
Try forcibly updating the offline store to tombstone, and then use remove-tombstone.
curl -X POST http://{pd_ip}:2379/pd/api/v1/store/${store_id}/state?state=Tombstone
Observe “region_count”: whether there is a decreasing trend until it becomes 0 and turns into a tombstone state.
You can modify the following parameters to accelerate the process, adjust x based on the current values according to the load:
config set region-schedule-limit x
config set replica-schedule-limit x
config set leader-schedule-limit x
config set max-pending-peer-count x
config set max-snapshot-count x
config set merge-schedule-limit x
Ok, it’s slowly decreasing now. I’ll observe it again tomorrow.
Okay, it’s slowly decreasing now. I’ll observe it again tomorrow.
It is estimated that it has not been completely taken offline yet. The region needs to migrate. You can check the monitoring panel, and it will become a tombstone after it is completely offline.
What you mentioned, curl -X POST http://{pd_ip}:2379/pd/api/v1/store/${store_id}/state?state=Tombstone, the original poster can try it, but I remember this operation is not very feasible. I used it before (the reason is that the version of the PD API no longer supports modifying to tombstone, it can only be modified to up and offline):
There are still quite a few anomalies caused by such forced scale-in operations. I have already forgotten how to handle it, sweat…
- Offline: When a TiKV Store is manually taken offline through PD Control, the Store will enter the Offline state. This state is merely an intermediate state for the Store going offline. In this state, the Store will move all its Regions to other Up state Stores that meet the relocation conditions. Once the Store’s leader count and region count (retrieved via PD Control) both show 0, the Store will transition from Offline to Tombstone state. During the Offline state, it is prohibited to shut down the Store service or its physical server. During the offline process, if there are no other target Stores in the cluster that meet the relocation conditions (for example, there are not enough Stores to continue meeting the cluster’s replica requirements), the Store will remain in the Offline state.
I encountered a situation where a node’s disk failed, and it eventually became Tombstone.
What did you do to turn it into a Tombstone?
This API is no longer usable.
It broke at night, but the data was already balanced during the day and it turned into a Tombstone on its own.
Yes, I just checked after one day, the status also changed to Tombstone, and then I executed remove-tombstone to completely remove it from PD.
From our two encounters, it can be seen that there is no need to worry about a single TiKV node failure, as TiDB will self-repair internally.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
|
username: LKLK
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: