loadRegion from PD failed: Irrecoverable error occurred when using TiKV raw client

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: loadRegion from PD failed 使用 tikv raw client 出现不可恢复报错

| username: TiDBer_d9ncNTPX

Usage scenario: Deployed pd + tikv as pure kv, using go-client’s rawkv to connect to pd for reading and writing. After using the same client for a period of time, an error “loadRegion from PD failed, key: ‘395F3130303130343235343932395F30303030303030313431’, err: rpc error” occurs, leading to a timeout and inability to automatically recover. Creating a new client does not solve the issue, and only restarting the instance works.
While the error occurs, other instances connecting to pd are normal, and the pd status is also normal. What could be the reason for this?

| username: xfworld | Original post link

Check the PD logs for any anomalies.
Also, the TiKV logs need to be checked.

If you can find any abnormal logs, you can pass the key information here, and I’ll help you analyze it.

However, you need to provide the version number first.

| username: shigp_TIDBER | Original post link

Please provide the logs for PD and TiKV. This kind of issue can only be analyzed through the logs.

| username: TiDBer_d9ncNTPX | Original post link

Version 6.5.8 has been supplemented.

| username: TiDBer_d9ncNTPX | Original post link

There are no error logs. After enabling debug logs, I see many gRPC errors: [2024/04/19 18:03:20.258 +08:00] [DEBUG] [lease.go:115] [“failed to receive lease keepalive request from gRPC stream”] [error=“rpc error: code = Canceled desc = context canceled”], but I’m not sure if they are related to the request.

| username: TiDBer_d9ncNTPX | Original post link

I found the error in the client, it should be caused by the GetRegion request. However, I tried calling GetPDClient().GetRegion directly before put and it worked fine, which is very strange.

| username: xfworld | Original post link

What configuration is the cluster built with?

| username: TiDBer_JUi6UvZm | Original post link

Are there no error logs for both PD and TiDB server?

| username: 洪七表哥 | Original post link

Still need to start by looking at the logs.