TiDB Request for Truncated KV

[translator_bot]

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 请求已经缩掉的kv

| username: Holland

---

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
tidb 4.0.14. I removed a KV node, and it is no longer visible in pd-ctl. Why is TiDB still requesting this offline KV?

[Reproduction Path] What operations were performed to cause the issue
[Encountered Issue: Symptoms and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

[translator_bot]

| username: weixiaobing | Original post link

---

When was the scale-down completed? Use tiup cluster display to check.

[translator_bot]

| username: Holland | Original post link

---

Did not use tiup

[translator_bot]

| username: Holland | Original post link

---

It looks like 13:56.

[translator_bot]

| username: weixiaobing | Original post link

---

So how did you scale down?

[translator_bot]

| username: Holland | Original post link

---

First, evict the store’s leader, then execute pd-ctl store delete 5016. After waiting for the KV to become a tombstone, execute remove tombstone.

[translator_bot]

| username: weixiaobing | Original post link

---

This approach may result in the cluster information not being updated, so TiDB will still make requests, which can take a long time. It is recommended to use TiUP for scaling in and out: 使用 TiUP 扩容缩容 TiDB 集群 | PingCAP 归档文档站

[translator_bot]

| username: h5n1 | Original post link

---

What specific command was executed, and what was the error reported? It seems like the region information in the PD cache hasn’t been updated.

[translator_bot]

| username: Holland | Original post link

---

pd-ctl

1. scheduler add evict-leader-scheduler-5106
2. Wait for the leader to become 0
3. store delete 5016
4. Wait for the store to become tombstone
5. store remove-tombstone

[translator_bot]

| username: Holland | Original post link

---

I have restarted both PD and TiDB. It’s useless.

[translator_bot]

| username: Billmay表妹 | Original post link

---

Check out this article by Binbin.

[translator_bot]

| username: Holland | Original post link

---

I replaced all the TiDB nodes through scaling up and down, and then restored them. Also, when the TiDB layer encounters a region miss during a query, it will re-fetch the region information from PD, right? Could it be that the fetch was unsuccessful? If it was successful, why does it still request the KV that has already been removed next time?

[translator_bot]

| username: Billmay表妹 | Original post link

---

Sure, please provide the text you need translated.

[translator_bot]

| username: Holland | Original post link

---

But on my side, I waited until it became a tombstone, rather than using --force to forcibly take it offline.

[translator_bot]

| username: Holland | Original post link

---

The monitoring also shows that it waits until the store region count drops to zero.

[translator_bot]

| username: Holland | Original post link

---

Could it be that there is an unaddressed bug related to the region cache?

[translator_bot]

| username: 会飞的土拨鼠 | Original post link

---

Have any corresponding changes been made in the configuration file? Check if you need to comment out a kv node in the configuration file.

[translator_bot]

| username: Billmay表妹 | Original post link

---

Scaling Issues - Column Articles

• TiFlash 6.0 on K8s Scaling and New Features Practice

• Pitfalls Encountered When Scaling TiKV Nodes

• TiDB Operator Scaling and Cluster Management

• Three Steps to Handle Abnormal TiKV Scaling Down

• How to Solve the Issue of TiKV Not Scaling Down?

• Rolling Upgrade Required for Entire Cluster After Scaling PD Nodes in TiDB-v3.0 Deployed via Docker

• TiFlash 6.0 on K8s Scaling and New Features Practice

• Pain Points Solved by TiDB Amid the Wave of Domestic Production

• Steps to Scale Down TiKV in Production Environment TiDB Cluster

• TiDB Monitoring Node Scaling Operations (Whether to Retain Historical Data)

Scaling Issues - Technical Q&A

• SSH Error During TiDB Scaling, Prompting Inaccessibility, but SSH is Actually Fine.

• Region Imbalance After TiKV Scaling

• Consultation on Scaling Issues After Forcibly Scaling Down TiFlash with --force

• Disk Alert Scaling for TiKV (Achieved by Scaling Up First and Then Scaling Down)

• Node Remains in Pending Offline State After TiDB Scaling Down

• TiCDC GC-TTL and Scaling Issues

• PD Normal Scaling Down, Database Unusable, Truncate Table Error: 1105 - Failed to Get Old Label Rules from PD

• TiUP Scaling (Node Migration via Scaling)

• Cannot Scale Down TiKV by Modifying TiDBCluster Replica

• Error When Scaling Up TiKV Nodes

• Store Region Score Fluctuates After Scaling TiKV Nodes from 3 to 5

• TiKV Scaling Up and Down Issues

• Unable to Delete Scaled Down Nodes Using Store Delete After Forcibly Scaling Down

• Regarding TiFlash Scaling Down Issues

• Consultation on Scale-In Scaling Down Command Issues

• Will Dashboard Automatically Migrate to New PD Nodes During Scaling?

• Can Automatic Balancing Be Disabled During TiDB Scaling?

• TiKV Scaling Up and Down Issues

• Cluster Information Error After TiKV Scaling Up and Down, Still Accessing Offline TiKV Nodes

• Failed to Scale Up TiFlash

• TiDB Storage Scaling

• Reload Error After Scaling Down TiKV When Migrating from 3.0 to 4.0

• TiKV Scaling Issues

• Mutual Scaling Issues Between Two Clusters

• Error When Starting TiKV Node After Scaling Down in TiDB 5.0.0

• Leader Loss Issue During TiKV Scaling Down, Any Solutions?

• TiKV Scaling Down Stuck in Pending State

• TiUP Scaling Up or Down

• Error When Scaling Down Nodes via TiUP

Scaling Issues - Documentation/SOP

• Using TiUP to Scale TiDB Cluster

• TiDB Operator Scaling and Cluster Management

• Scaling Down Nodes

• Scaling Down Cluster

• Horizontal Scaling

• SOP Series 08: Online Cluster Scaling Operations and Key Points

• Scaling Down TiFlash Nodes

• Scaling Down Pump Nodes

• Scaling Down and Fault Recovery

• Scaling TiDB Cluster

• Using TiUP to Scale TiDB Cluster

You can refer to some related content on scaling issues~