When using tiup to scale in, it remains in the Pending Offline state instead of transitioning to the tombstone state

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用tiup缩容的时候一直没有转化为tombstone状态,而是一直处于Pending Offline状态

| username: TiDBer_微风轻吟

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 7.5.1
[Reproduction Path] [Scale-in Documentation] (使用 TiUP 扩容缩容 TiDB 集群 | PingCAP 文档中心)
Execute the command tiup cluster scale-in --node 10.0.1.5:20160 as per the documentation
It remains in the offline state and never enters the deletable tombstone state
[Attachment: Screenshot/Log/Monitoring]

| username: caiyfc | Original post link

First, check the monitoring to confirm whether all leaders and regions have been migrated. If they have, use pd ctl to check the corresponding store and see if there are any regions left. Manually migrate them. When the store has no regions left, it will become a tombstone.

| username: Jellybean | Original post link

OP, how many TiKV nodes does your cluster have?
If there are only 3 TiKV nodes, this phenomenon will occur because there are no extra nodes to migrate data.
If there are more than 3 TiKV nodes, further analysis is needed based on the amount of data on a single node and the cluster load.

| username: DBAER | Original post link

Check the TiKV logs and look at the region monitoring records.

| username: TiDBer_微风轻吟 | Original post link

How should I solve this with only three TiKV nodes, master?

| username: zhang_2023 | Original post link

The region hasn’t finished migrating, right?

| username: Soysauce520 | Original post link

Expand first, then shrink.

| username: TiDBer_微风轻吟 | Original post link

Yes, it hasn’t moved at all. When using the store to check, this region hasn’t moved :sob:

| username: TiDBer_微风轻吟 | Original post link

In theory, this seems to solve the problem, but there are no extra machines available. Is there any other solution, expert?

| username: TiDBer_微风轻吟 | Original post link

When using pd ctl to check the store, the regions inside haven’t moved :sob:

| username: Soysauce520 | Original post link

You can expand capacity by changing the directory and port on the same machine, but it will be difficult to manage later.

| username: TiDBer_微风轻吟 | Original post link

Okay, thank you, I’ll give it a try.

| username: TiDBer_微风轻吟 | Original post link

Thank you, the regions are already decreasing.

| username: zhh_912 | Original post link

Check if the migration is truly complete.

| username: 舞动梦灵 | Original post link

With only three TiKV nodes, you cannot scale down. You can only scale down after scaling up.

| username: TiDBer_微风轻吟 | Original post link

Yes, thank you, the issue has been resolved.