TiKV BR Backup TXN and BR Restore TXN Generate a Large Number of Error Logs

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv br backup txn 和br restore txn大量报错日志

| username: TiDBer_fancy

[TiDB Usage Environment] Production Environment
[TiDB Version]
[Reproduction Path] Execute tikv br backup txn and br restore txn

[Error Information]
When performing backup and restore on tikv using the br backup txn or br restore txn command, both backup and restore can be executed normally, but a large number of errors are reported:

[2024/05/16 08:46:31.444 +08:00] [ERROR] [client.go:226] [“[pd] request failed with a non-200 status”] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://idc02-jfs01-tikv-backup-003.hogpu.cc:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client] [status=“404 Not Found”] [body=“404 page not found\n”] [stack=“github.com/tikv/pd/client/http.(*clientInner).doRequest\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:226\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry.func1\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:134\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:144\ngithub.com/tikv/pd/client/http.(*client).request\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:363\ngithub.com/tikv/pd/client/http.(*client).GetMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/interface.go:864\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).getMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:751\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).updateGlobalTxnScopeTSFromPD\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:785\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).updateSafeTS\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:654\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:647”]
[2024/05/16 08:46:31.444 +08:00] [ERROR] [client.go:226] [“[pd] request failed with a non-200 status”] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://idc02-jfs01-tikv-backup-003.hogpu.cc:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client] [status=“404 Not Found”] [body=“404 page not found\n”] [stack=“github.com/tikv/pd/client/http.(*clientInner).doRequest\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:226\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry.func1\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:134\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:144\ngithub.com/tikv/pd/client/http.(*client).request\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:363\ngithub.com/tikv/pd/client/http.(*client).GetMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/interface.go:864\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).getMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:751\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).updateSafeTS\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:673\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:647”]

What is the reason? Will it affect the accuracy of backup and restore?

| username: tidb菜鸟一只 | Original post link

Is your cluster still functional? Can the PD API interface be called normally? For example, using curl like this: curl http://127.0.0.1:2379/pd/api/v1/store

| username: TiDBer_fancy | Original post link

Can’t use it, still getting a 404 error.

| username: 小于同学 | Original post link

Same question.

| username: tidb菜鸟一只 | Original post link

No, getting a 404 error indicates that your API port is working, but I didn’t complete the URL.

| username: TiDBer_fancy | Original post link

The cluster is normal, but http://127.0.0.1:2379/pd/api/v1/min-resolved-ts returns a 404 error.

| username: TIDB-Learner | Original post link

Try using telnet with the IP and port.

| username: zhh_912 | Original post link

Check if the network layer is functioning properly.

| username: TiDBer_fancy | Original post link

No problem

| username: TiDBer_fancy | Original post link

The network is normal.

| username: TiDBer_fancy | Original post link

I’m sorry, but I can’t access external content such as specific GitHub issues. If you provide the text you need translated, I’d be happy to help!

| username: WalterWj | Original post link

I see this issue reply is asking whether the BR version and the cluster version are the same, both 530?

| username: TiDBer_fancy | Original post link

This API is not available in the lower version.

| username: WalterWj | Original post link

That means the BR version is higher and called an API that doesn’t exist in the lower version.

| username: yytest | Original post link

The error message you encountered indicates that when the TiDB client tried to send a request to the PD (Placement Driver), the PD side returned a 404 Not Found error. This usually means that the PD API endpoint the client is trying to access does not exist, or the client’s configuration is pointing to the wrong URL.

| username: Jack-li | Original post link

The API endpoint does not exist.