BR Backup Error: [pd] fetch pending tso requests error

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: BR备份报错：[pd] fetch pending tso requests error

| username: TiDBer_Lee

【TiDB Usage Environment】Production Environment / Testing / Poc
【TiDB Version】v5.4.3
【Reproduction Path】The TiDB cluster was upgraded from v5.1.4 to v5.4.3, and it was found that all backups would report errors. Data restoration can succeed, but this error is also reported.
【Encountered Problem: Phenomenon and Impact】
The backup error is as follows:
[2023/05/18 09:36:26.756 +00:00] [INFO] [schema.go:137] [“backup checksum”] [take=1m8.099528156s]
[2023/05/18 09:36:26.756 +00:00] [INFO] [metafile.go:562] [“write metas finished”] [type=schema]
[2023/05/18 09:36:26.756 +00:00] [INFO] [metafile.go:602] [“finish the write metas”] [item=727] [type=schema] [costs=1m8.099578709s]
[2023/05/18 09:36:26.757 +00:00] [INFO] [metafile.go:622] [“save backup meta”] [size=223283]
[2023/05/18 09:36:26.864 +00:00] [INFO] [validate.go:78] [“checksum success”] [db=discovery_plus] [table=patsnap_master_entity_discovery_techindex_20230518]
[2023/05/18 09:36:26.864 +00:00] [ERROR] [client.go:752] [“[pd] fetch pending tso requests error”] [dc-location=global] [error=“[PD:client:ErrClientGetTSO]context canceled: context canceled”] [stack=“github.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:752”]
[2023/05/18 09:36:26.864 +00:00] [INFO] [client.go:666] [“[pd] exit tso dispatcher”] [dc-location=global]
[2023/05/18 09:36:26.865 +00:00] [ERROR] [client.go:752] [“[pd] fetch pending tso requests error”] [dc-location=global] [error=“[PD:client:ErrClientGetTSO]context canceled: context canceled”] [stack=“github.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:752”]
[2023/05/18 09:36:26.865 +00:00] [INFO] [client.go:666] [“[pd] exit tso dispatcher”] [dc-location=global]
[2023/05/18 09:36:26.865 +00:00] [INFO] [collector.go:68] [“Table backup success summary”] [total-ranges=726] [ranges-succeed=726] [ranges-failed=0] [backup-fast-checksum=608.943µs] [backup-checksum=1m8.099548023s] [backup-total-ranges=2] [backup-total-regions=463] [total-take=5m39.143832204s] [total-kv-size=37.54GB] [average-speed=110.7MB/s] [backup-data-size(after-compressed)=15.27GB] [Size=15273290208] [BackupTS=441555942879002627] [total-kv=252076760]

Restore error:
[2023/05/18 09:54:59.720 +00:00] [INFO] [domain.go:716] [“domain closed”] [“take time”=11.843483ms]
[2023/05/18 09:54:59.720 +00:00] [INFO] [manager.go:277] [“failed to campaign”] [“owner info”=“[stats] /tidb/stats/owner ownerManager 16582131-38c0-4ad2-be77-3c91d0b7c702”] [error=“rpc error: code = Unavailable desc = transport is closing”]
[2023/05/18 09:54:59.720 +00:00] [INFO] [manager.go:249] [“etcd session is done, creates a new one”] [“owner info”=“[stats] /tidb/stats/owner ownerManager 16582131-38c0-4ad2-be77-3c91d0b7c702”]
[2023/05/18 09:54:59.721 +00:00] [INFO] [manager.go:277] [“failed to campaign”] [“owner info”=“[stats] /tidb/stats/owner ownerManager 16582131-38c0-4ad2-be77-3c91d0b7c702”] [error=“rpc error: code = Unavailable desc = transport is closing”]
[2023/05/18 09:54:59.721 +00:00] [INFO] [manager.go:249] [“etcd session is done, creates a new one”] [“owner info”=“[stats] /tidb/stats/owner ownerManager 16582131-38c0-4ad2-be77-3c91d0b7c702”]
[2023/05/18 09:54:59.721 +00:00] [ERROR] [client.go:752] [“[pd] fetch pending tso requests error”] [dc-location=global] [error=“[PD:client:ErrClientGetTSO]context canceled: context canceled”] [stack=“github.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:752”]
[2023/05/18 09:54:59.721 +00:00] [INFO] [client.go:666] [“[pd] exit tso dispatcher”] [dc-location=global]
[2023/05/18 09:54:59.722 +00:00] [ERROR] [client.go:752] [“[pd] fetch pending tso requests error”] [dc-location=global] [error=“[PD:client:ErrClientGetTSO]context canceled: context canceled”] [stack=“github.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:752”]
[2023/05/18 09:54:59.722 +00:00] [INFO] [client.go:666] [“[pd] exit tso dispatcher”] [dc-location=global]
[2023/05/18 09:54:59.722 +00:00] [INFO] [collector.go:68] [“Table restore success summary”] [total-ranges=727] [ranges-succeed=727] [ranges-failed=0] [split-region=4.13103895s] [restore-ranges=463] [total-take=4m28.764527919s] [restore-data-size(after-compressed)=15.27GB] [Size=15273290208] [BackupTS=441556253211885575] [total-kv=252076760] [total-kv-size=37.54GB] [average-speed=139.7MB/s]

This error is consistently reproducible when using the br backup tool v5.4.3 to back up the TiDB database (v5.4.3), but it is unclear if it has any impact.

2
| username: weixiaobing | Original post link

This should be a known issue. As long as there is a successful backup in the logs, it should be fine. [“Table backup success summary”]

3
| username: TiDBer_Lee | Original post link

Yes, this problem has existed for a long time and has not been fixed. In a production environment, the backup data is very large. If there is an issue during recovery, even just once, I would have to leave.

4
| username: Billmay表妹 | Original post link

The issue has been resolved in version 6.x. Please upgrade to version 6.x or above.

5
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.