Can a store that has entered tombstone status be re-added to the TiDB cluster?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 已进入 tombstone 的 store 是否可重加回 tidb cluster

| username: npes87184

To improve efficiency, please provide the following information. A clear problem description can help resolve the issue faster:
【TiDB Environment】k8s and tidb cluster

【TiDB Version】4.0.16

【Background】
During tkv scaling down, two stores had their leader count and region count stuck at 1 for over half an hour. We manually forced store 1002 and 1004 into tombstone state using:
curl -X POST ‘http://{pd_ip}:{pd_port}/pd/api/v1/store/{store_id}/state?state=Tombstone’

Afterwards, we discovered that region 10353493 was in an abnormal state, with one peer as a learner and the other two peers inaccessible due to their stores being set to tombstone. The Tidb cluster ready status is true. We have tried to bring store 1002 back up using:
curl -X POST ‘http://{pd_ip}:{pd_port}/pd/api/v1/store/{store_id}/state?state=Up’

We also attempted to expand the tikv replica again to create a new tikv node and mount the corresponding store 1002, but it seems to have no effect. The statefulset of tikv has not been modified accordingly.

Abnormal region status: by using ./pd-ctl region 10353493

{
  "id": 10353493,
  "start_key": "7480000000000000FF175F728000000000FF15D0A10000000000FA",
  "end_key": "7480000000000000FF175F728000000000FF15D1720000000000FA",
  "epoch": {
    "conf_ver": 269722,
    "version": 1216
  },
  "peers": [
    {
      "id": 10385018,
      "store_id": 1002
    },
    {
      "id": 10389925,
      "store_id": 1004
    },
    {
      "id": 10344516,
      "store_id": 1013,
      "is_learner": true
    }
  ],
  "leader": {
    "id": 10385018,
    "store_id": 1002
  },
  "down_peers": [
    {
      "peer": {
        "id": 10344516,
        "store_id": 1013,
        "is_learner": true
      },
      "down_seconds": 300
    }
  ],
  "pending_peers": [
    {
      "id": 10344516,
      "store_id": 1013,
      "is_learner": true
    }
  ],
  "written_bytes": 0,
  "read_bytes": 0,
  "written_keys": 0,
  "read_keys": 0,
  "approximate_size": 1,
  "approximate_keys": 0
}

【Business Impact】
Using SQL syntax select * from mysql.stats_histograms and select * from information_schema.tables; results in the following error:
(1105, ‘no available peers, region: {id:10353493 start_key:“t\200\000\000\000\000\000\000\027_r\200\000\000\000\000\025\320\241” end_key:“t\200\000\000\000\000\000\000\027_r\200\000\000\000\000\025\321r” region_epoch:<conf_ver:269722 version:1216 > peers:<id:10385018 store_id:1002 > peers:<id:10389925 store_id:1004 > }’)

Backup using br fails.

【Question】
I would like to ask:

Since the current region has three peers in the states of tombstone, tombstone, and learner, is it possible to recover the region from tombstone, or directly remove the region?

Can the region be directly removed? (e.g., drop table)
Currently, the region belongs to the mysql.stats_histograms data, and losing it should not affect other data.

Thank you.

2
| username: h5n1 | Original post link

You can use pd-ctl to change the offline status to up status and then add it back. After it becomes tombstone, you can only prune and then scale-out to add it again.

3
| username: 长安是只喵 | Original post link

It cannot be directly reused; it must be expanded and data rebalanced.

4
| username: npes87184 | Original post link

So, may I ask, what methods are available to remove problematic regions?
Is using tiup cluster prune the only way to make the cluster perform the corresponding actions?

5
| username: songxuecheng | Original post link

You can try to restore the region.

6
| username: h5n1 | Original post link

Currently, there is only one Learner peer left. Without a leader, it cannot reach a normal state through pd-ctl add operator, etc. You can try the following methods:

  1. If the system is down, try using tikv-ctl unsafe-recover to recover the region with multiple replica failures.
  2. Use tikv-ctl recreate-region to rebuild the region.
  3. Use tikv-ctl tombstone to discard the region.

Methods 2 and 3 will result in data loss. The mysql.stats_histograms belong to statistical information, and it is uncertain whether subsequent processing will affect the loading of statistical information.

7
| username: npes87184 | Original post link

May I ask, is tikv-ctl also unable to operate on tombstore?

Additionally, I have a silly question: how do you shut down a tikv node? Is it done by changing the weight of the store to 0 through pd-ctl and then deleting it?

Thanks.

8
| username: h5n1 | Original post link

tikv-ctl cannot operate tombstone store. The pe-ctl delete store is an offline process, during which regions/leaders are transferred to other TiKV instances. Once all transfers are completed, it will become a tombstone store. Doesn’t your version have tiup? Try to use tiup scale-in for scaling down. Additionally, after it becomes a tombstone, use pd-ctl -u http://pd_ip:2379 store remove-tombstone or curl -X DELETE pd-addr:port/pd/api/v1/stores/remove-tombstone to clean it up.

9
| username: songxuecheng | Original post link

I remember that scaling down in k8s involves changing the replica count, but the scaling down is done in order.

10
| username: h5n1 | Original post link

Oh, I’m not very familiar with k8s operations, and indeed there’s no tiup.

11
| username: songxuecheng | Original post link

Have the two TiKV pods been scaled down now?
If the scaling down is complete, check the regions.

image
image1380×666 77.7 KB

12
| username: npes87184 | Original post link

Yes, the scale-in has been completed. After the store entered the tombstone state, the TiKV node was also removed.

After using pd-ctl to check the region,

except for the aforementioned region 10353493 which has been in pending peer and down peer status,

the status of the other peers is only temporary.

13
| username: songxuecheng | Original post link

If there is only one down peer on the already scaled-down KV pod, you can remove the peer and let PD automatically compensate.

14
| username: npes87184 | Original post link

The main issue is that the current state of the three peers in this region is tombstone, tombstone, learner.

So I was curious to ask whether it is possible to recover the region from tombstone or directly remove the region.

15
| username: npes87184 | Original post link

I have tried using pd-ctl remove-tombstone multiple times to remove them, but these two tombstones still exist.

16
| username: songxuecheng | Original post link

It looks like all replicas have been lost. This requires destructive repair.
Refer to 专栏 - TiKV 多副本丢失以及修复实践 | TiDB 社区 for more information.

17
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.