TIFLASH [util.rs:463] invalid store ID 0, not found exception

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIFLASH [util.rs:463] invalid store ID 0, not found 异常

| username: TiDBer_ZhuaYQ4V

【TiDB Environment】Production/Test environment
【TiDB Version】v6.1.0
【Encountered Problem】
2022-09-08 16:22:33 (GMT+8) TiFlash 10.130.128.14:3930 [util.rs:463] [“request failed”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { code: 2-UNKNOWN, message: "invalid store ID 0, not found", details: }))”]
2022-09-08 16:22:33 (GMT+8) TiFlash 10.130.128.14:3930 [util.rs:463] [“request failed”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { code: 2-UNKNOWN, message: "invalid store ID 0, not found", details: }))”]
2022-09-08 16:22:33 (GMT+8) TiFlash 10.130.128.14:3930 [util.rs:472] [“reconnect failed”] [err_code=KV:PD:Unknown] [err=“Other("[components/pd_client/src/util.rs:309]: cancel reconnection due to too small interval")”]

【Reproduction Path】After completing TiKV expansion previously, everything was normal without such errors; after migrating a large table with over a billion rows to TiDB and creating TiFlash replicas, the errors started appearing. The current cluster environment is still usable but probably not very stable.
【Problem Phenomenon and Impact】

【Attachments】Related logs and monitoring (https://metricstool.pingcap.com/)

ERROR1
ERROR11350×876 78.2 KB

For questions related to performance optimization and fault troubleshooting, please download and run the script. Please select all and copy-paste the terminal output.

2
| username: AnotherCalvinNeo | Original post link

Could you please provide the tiflash_tikv.log log of the faulty node?

3
| username: Min_Chen | Original post link

Hello, is there any TiKV/TiFlash store with ID 0 in the cluster? Please provide the TiDB, PD, TiKV, and TiFlash logs, the output of pd-ctl store, and the status of table synchronization with the output of SELECT * FROM information_schema.tiflash_replica.

4
| username: alfred | Original post link

Is the status of TiFlash normal?

5
| username: 大飞飞呀 | Original post link

How was this issue finally resolved? We are encountering the same problem.