TiKV Node Store Remains in Offline Status and Cannot Be Decommissioned

translator_bot · June 22, 2024, 9:40pm

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv节点store一直处于offline状态，无法下线

| username: panqiao

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.1.0
[Reproduction Path] Operations that led to the issue
Since I needed to delete the PVCs of three TiKV nodes, I had to scale down TiKV to 0 and then scale it back up to 3, which resulted in the offline issue.
[Encountered Issue: Symptoms and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
I currently have five TiKV pods.

I want to scale down to 3, but when I do, it scales down 3 and 4, leaving the faulty 1 and 2.
The logs for 1 and 2 are:

The store information shows (the two faulty ones are down)

I executed the following command to bring them online:
curl -X POST http://127.0.0.1:2379/pd/api/v1/store/${store_id}/state?state=Up
The status changes to offline, but no matter how long I wait, the regions do not transfer.
Using pd-ctl store delete id also keeps it in an offline state.
I don’t know what to do. I want to restore the three TiKV nodes to normal. How should I proceed? Seeking help from all the experts.

translator_bot · June 22, 2024, 9:40pm

| username: WalterWj | Original post link

When scaling down, you need to consider the number of replicas and TiKV instances. If the number of nodes after scaling down is less than the number of replicas, the scaling down cannot be completed. The reason is that scaling down requires placing replicas on other nodes, and one node cannot have multiple replica data. This principle cannot be violated.
For scaling down on k8s, the k8s strategy is to start with the largest pod, and this strategy needs to be modified. Check this out: 增强型 StatefulSet 控制器 | PingCAP 文档中心

translator_bot · June 22, 2024, 9:40pm

| username: panqiao | Original post link

I modified the deletion policy, but it still deleted my 4.

translator_bot · June 22, 2024, 9:40pm

| username: WalterWj | Original post link

Shouldn’t the name be placed on the outer layer?

translator_bot · June 22, 2024, 9:40pm

| username: tidb菜鸟一只 | Original post link

There seems to be a problem with what’s written here, right?

translator_bot · June 22, 2024, 9:40pm

| username: WalterWj | Original post link

I saw the official website wrote it like this. Could it be that the official website is wrong? … I don’t have an environment to help test it …

translator_bot · June 22, 2024, 9:40pm

| username: yiduoyunQ | Original post link

Please provide the output of kubectl get tc {tc-name} -o yaml.
Please provide the output of the operator logs kubectl logs tidb-controller-manager-{xxxx}.
What is the version of the operator?

translator_bot · June 22, 2024, 9:40pm

| username: panqiao | Original post link

What bothers me the most right now is that I can’t remove the store information of 1 and 2 from my PD nodes. The leaders and regions on them just won’t transfer. If I don’t remove the information of these two stores, it will always be a potential risk.

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

Logs

I1213 07:29:34.989188 1 tikv_scaler.go:90] scaling in tikv statefulset tidb-cluster/pre-test-tikv, ordinal: 4 (replicas: 4, delete slots: )

E1213 07:29:34.997998 1 tikv_scaler.go:250] can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:34.998262 1 event.go:282] Event(v1.ObjectReference{Kind:“TidbCluster”, Namespace:“tidb-cluster”, Name:“pre-test”, UID:“b249cfb6-8ad0-4c50-897e-d21c0acee6f3”, APIVersion:“pingcap.com/v1alpha1”, ResourceVersion:“46472491”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedScaleIn’ can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:45.724813 1 tikv_scaler.go:90] scaling in tikv statefulset tidb-cluster/pre-test-tikv, ordinal: 4 (replicas: 4, delete slots: )
E1213 07:29:45.732652 1 tikv_scaler.go:250] can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:45.732806 1 event.go:282] Event(v1.ObjectReference{Kind:“TidbCluster”, Namespace:“tidb-cluster”, Name:“pre-test”, UID:“b249cfb6-8ad0-4c50-897e-d21c0acee6f3”, APIVersion:“pingcap.com/v1alpha1”, ResourceVersion:“46472491”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedScaleIn’ can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:45.754481 1 tidbcluster_control.go:69] TidbCluster: [tidb-cluster/pre-test] updated successfully
I1213 07:29:45.865868 1 tikv_scaler.go:90] scaling in tikv statefulset tidb-cluster/pre-test-tikv, ordinal: 4 (replicas: 4, delete slots: )
E1213 07:29:45.875868 1 tikv_scaler.go:250] can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:45.876047 1 event.go:282] Event(v1.ObjectReference{Kind:“TidbCluster”, Namespace:“tidb-cluster”, Name:“pre-test”, UID:“b249cfb6-8ad0-4c50-897e-d21c0acee6f3”, APIVersion:“pingcap.com/v1alpha1”, ResourceVersion:“46474051”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedScaleIn’ can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:51.843729 1 tikv_scaler.go:90] scaling in tikv statefulset tidb-cluster/pre-test-tikv, ordinal: 4 (replicas: 4, delete slots: )
E1213 07:29:51.851744 1 tikv_scaler.go:250] can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:51.852085 1 event.go:282] Event(v1.ObjectReference{Kind:“TidbCluster”, Namespace:“tidb-cluster”, Name:“pre-test”, UID:“b249cfb6-8ad0-4c50-897e-d21c0acee6f3”, APIVersion:“pingcap.com/v1alpha1”, ResourceVersion:“46474051”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedScaleIn’ can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too
I1213 07:29:51.869532 1 tidbcluster_control.go:69] TidbCluster: [tidb-cluster/pre-test] updated successfully
I1213 07:29:51.965092 1 tikv_scaler.go:90] scaling in tikv statefulset tidb-cluster/pre-test-tikv, ordinal: 4 (replicas: 4, delete slots: )
E1213 07:29:51.972914 1 tikv_scaler.go:250] can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too

I1213 07:29:51.973294 1 event.go:282] Event(v1.ObjectReference{Kind:“TidbCluster”, Namespace:“tidb-cluster”, Name:“pre-test”, UID:“b249cfb6-8ad0-4c50-897e-d21c0acee6f3”, APIVersion:“pingcap.com/v1alpha1”, ResourceVersion:“46474101”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedScaleIn’ can’t scale in TiKV of TidbCluster [tidb-cluster/pre-test], cause the number of up stores is equal to MaxReplicas in PD configuration(3), and the store in Pod pre-test-tikv-4 which is going to be deleted is up too

Version: 1.3.9

translator_bot · June 22, 2024, 9:41pm

| username: WalterWj | Original post link

You can manually clean this up by entering the PD pod and using pd-ctl to clean it. Use store delete to manually delete it. You can refer to the official website for the usage of pd-ctl.

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

I tried it, boss. After I used delete, it changed from down status to offline status, and then it stayed like that. After a few days, its region still hasn’t moved.

translator_bot · June 22, 2024, 9:41pm

| username: WalterWj | Original post link

Are the remaining nodes enough to make up the replicas?

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

Are you referring to the number of region replicas? If so, how can I check the number of region replicas?

translator_bot · June 22, 2024, 9:41pm

| username: WalterWj | Original post link

If it hasn’t been changed, it’s 3. After scaling down, the remaining TiKV pods must be >= 3.

translator_bot · June 22, 2024, 9:41pm

| username: yiduoyunQ | Original post link

Now we need to first schedule the leaders on store-4 tikv-1 and store-5 tikv-2 to other up tikvs (not sure why it didn’t auto-schedule). After the scheduling is complete, we can continue with the operations.

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

I also met this requirement. When I first asked this question, didn’t I have five KV pods, three good ones and two bad ones? It just wouldn’t transfer.

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

I have also tried several methods in the official documentation. Currently, it cannot be directly set to tombstone status; we can only wait for it to schedule itself. However, I maintained the state of 5 pods, with 3 running, for 2 days, and it still hasn’t transferred.

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

Is there any way to clear the store information?

translator_bot · June 22, 2024, 9:41pm

| username: panqiao | Original post link

Moreover, what’s strange is that there isn’t a single leader on my other two good pods.

translator_bot · June 22, 2024, 9:41pm

| username: yiduoyunQ | Original post link

A brief explanation of the current situation is as follows:

These 3 are the original TiKV:
store-1 tikv-0 up leader count 4
store-4 tikv-1 down leader count 8
store-5 tikv-2 down leader count 2

These 2 are auto failover TiKV:
store-72180 tikv-3 auto failover up leader count 0
store-72223 tikv-4 auto failover up leader count 0

Scaling down can only start with auto failover TiKV, therefore:

Please manually schedule the leader first to ensure that the original store-4 and store-5 have no leaders.
Use pd-ctl store delete to remove store-4 and store-5, ensuring their status changes from down to tombstone.
Delete the PVC and pod of store-4 and store-5 (without data). The operator will automatically reschedule them, adding them with empty data. Wait for region scheduling to balance and ensure the new TiKV (store ID may not be 4 and 5) status is up.
Refer to Kubernetes 上的 TiDB 集群故障自动转移 | PingCAP 文档中心 to restore and scale down store-72180 and store-72223.
(Optional) Use pd-ctl store remove-tombstone to clean up the tombstone TiKV from step 3.