After Decommissioning a Node in TiFlash 5.4, MPP Queries Still Look for the Old StoreID

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiflash 5.4 下线节点后，mpp查询 仍然找旧的storeid

| username: foxchan

[TiDB Usage Environment] Production Environment
[TiDB Version] v5.4.2
[Reproduction Path] Decommission a TiFlash and enable MPP
[Encountered Problem: Problem Phenomenon and Impact]
Executing SQL frequently reports errors, store ID is the decommissioned TiFlash, suspected to be cache issue.
Checking peer is empty,
region --jq=".regions | {id: .id, peer_stores: [.peers.store_id] | select(any(.==3918054)) }

Error as follows:
ERROR 1105 (HY000): other error for mpp stream: DB::Exception: Exchange receiver meet error : DB::Exception: get store failed: 2: invalid store ID 3918054, not found

2
| username: songxuecheng | Original post link

You can temporarily turn off MPP first, and then check the PD store status.

3
| username: foxchan | Original post link

TiFlash has already removed the tombstore, and it has been removed for several days, but it is still reporting.

4
| username: songxuecheng | Original post link

Have you switched the PD leader?

5
| username: foxchan | Original post link

I haven’t switched.

6
| username: songxuecheng | Original post link

Did you not restart PD after it went offline?

7
| username: foxchan | Original post link

No.

8
| username: songxuecheng | Original post link

Have you performed the upgrade operation?

9
| username: foxchan | Original post link

Upgrading TiFlash from 5.4 to 6.1, the business relies on TiFlash, so TiFlash is prioritized for the upgrade. However, TiFlash can only be upgraded online and offline…

10
| username: songxuecheng | Original post link
  1. To take TiFlash offline, you need to clear the replicas first before taking it offline. This depends on your specific operations.
  2. For the upgrade, first disable MPP and then proceed with the upgrade. You don’t need to scale TiFlash up or down.
  3. Regarding your current issue, I suspect it might be related to Removed tombstone stores show again if transfer pd leader during scale in · Issue #4941 · tikv/pd · GitHub. You can take a look at this.
11
| username: foxchan | Original post link

It seems that tidb 5.4.3 has fixed this bug. I’ll upgrade to 6.1 first.

12
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.