How to Handle the State of Pending Offline After Scaling Down

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 缩容后一直处于Pending Offline状态怎么处理

| username: 月明星稀

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.0
[Reproduction Path] tiup cluster scale-in test_group --node 1.1.1.5:20160 -y
[Encountered Problem: Problem Phenomenon and Impact] Stuck in Pending Offline status

| username: Christophe | Original post link

Is the number of nodes less than the number of replicas? If so, once the new nodes have finished synchronizing the regions, the scaled-down nodes will enter the tombstone state, and then you can remove them.

| username: 月明星稀 | Original post link

Originally there were 5 nodes, so it shouldn’t be due to a lack of replicas, right?

| username: 有猫万事足 | Original post link

Check the Grafana monitoring to see if regions are being scheduled, and you’ll know the progress. If there’s a lot of data, it might be a bit slow.

| username: Christophe | Original post link

Let’s wait and see if the region migration is complete. Once it’s done, it should be fine.

| username: 月明星稀 | Original post link

Are there any other logs to check besides Grafana?

| username: tidb菜鸟一只 | Original post link

Execute pdctl stores to check the number of regions, it should not have finished migrating.

| username: Kongdom | Original post link

First of all, this situation is normal.
Then, it is recommended to check the PD monitoring in Grafana to see if the region migration is complete.

| username: 有猫万事足 | Original post link

It’s fine to see the scheduling in the PD logs, but it’s really not as convenient as Grafana.

| username: 月明星稀 | Original post link

Are you looking at the PD leader logs?

| username: 月明星稀 | Original post link

Can it be viewed using tiup?

| username: 月明星稀 | Original post link

Currently, the cluster data is not much, the data file is 1GB, and after waiting for more than an hour, it is still in Pending Offline status. If we use --force directly online, will it cause data loss?

| username: h5n1 | Original post link

| username: 像风一样的男子 | Original post link

In the Grafana monitoring, you can see the remaining number of regions on this node.

| username: tidb菜鸟一只 | Original post link

Use tiup ctl:v6.5.0 pd -u http://10.10.10.14:2379 -i to specify your PD node to enter pdctl, then input store.

| username: zxgaa | Original post link

Wait a bit longer.

| username: Kongdom | Original post link

Yes, if you use --force before the data migration is complete, it will cause data loss. You can refer to the three-step solution mentioned above.

| username: andone | Original post link

It should be in the scheduling.

| username: oceanzhang | Original post link

Check the logs to see if any resources are stuck.

| username: oceanzhang | Original post link

Has it been resolved?