When using TiKV alone, a KeyPadding error occurs when scanning a certain range

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 单独使用TIKV时,scan某个range时出现KeyPadding错误

| username: ethanlee

【TiKV Usage Environment】Production Environment
【TiKV Version】v7.4.0
【Reproduction Path】Uncertain when the issue started, suspect it might be related to a previous scaling operation.
【Encountered Issue: Symptoms and Impact】
Using the Go client (v2.0.7), occasionally a txnkv scan request returns an error (tikv aborts txn: Error(Txn(Error(Codec(KeyPadding))))).

Upon investigating the corresponding server logs:
[ERROR] [errors.rs:467] [“txn aborts”] [err_code=KV:Codec:BadPadding] [err=Error(Txn(Error(Codec(KeyPadding))))]

After careful debugging, it was found that the error only occurs when the scan’s startKey and endKey fall within a specific range of a certain region.
For example: region id = 10000, region’s start_key is 1a2a… and end_key is 1a2f… When the scan specifies [1a2b… to 1a2c…] or [1a2e to 1a2f], everything works fine, but when specifying [1a2d… to 1a2e…], the above error is triggered. It seems like the data in this range is corrupted.

Here is some pseudo-code in Golang:

startKey := "xxx"
endKey := "yyy"

client, _ := txnkv.NewClient([]string{"addr:2379"})
txn, _ := client.Begin()
it, err := txn.Iter(startKey, endKey)

if err != nil {
	log.Printf("tx.Iter err:%v", err) // When the issue occurs, it reports: tikv aborts txn: Error(Txn(Error(Codec(KeyPadding))))
	return
}
log.Println("ok")
| username: Fly-bird | Original post link

Combine on the left and open on the right, try reducing a number on the right side.

| username: 随缘天空 | Original post link

The RemoteStopped error indicates that the remote node has stopped. This could be due to network issues, node failures, or other reasons that prevent establishing a connection or communication with the remote node. Manually copy the following link into your browser to see if you can access it: tikv/src/server/service/kv.rs at v7.4.0 · tikv/tikv · GitHub. It is likely caused by different access conditions to GitHub.

| username: ethanlee | Original post link

@随缘天空 This link is accessible. RemoteStopped is quite strange because the node where this region is located is up and running, and the ranges before and after it are accessible (both are also on this region). So I suspect that the data is corrupted. I will continue to look at the source code.

| username: heiwandou | Original post link

There are bad sectors.

| username: 随缘天空 | Original post link

Hmm, check if anyone has manually deleted certain storage directories or files on the server.