Error When Backing Up TiDB to MinIO

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb备份到minio出错

| username: TiDBer_28PtrpL9

[TiDB Usage Environment] Production Environment
[TiDB Version] tidb-operator v1.1.12
Backing up to MinIO storage within the local network, followed the official documentation, encountered error logs

Create rclone.conf file.
/tidb-backup-manager backup --namespace=aimo-tidb --backupName=aimo-tidb-backup-s3-16 --tikvVersion=v5.0.1
I1118 18:45:31.245506 9 backup.go:71] start to process backup aimo-tidb/aimo-tidb-backup-s3-16
I1118 18:45:31.326494 9 backup_status_updater.go:84] Backup: [aimo-tidb/aimo-tidb-backup-s3-16] updated successfully
E1118 18:45:31.335421 9 backup_status_updater.go:87] Failed to update backup [aimo-tidb/aimo-tidb-backup-s3-16], error: Operation cannot be fulfilled on backups.pingcap.com “aimo-tidb-backup-s3-16”: the object has been modified; please apply your changes to the latest version and try again
I1118 18:45:31.383023 9 backup_status_updater.go:84] Backup: [aimo-tidb/aimo-tidb-backup-s3-16] updated successfully
I1118 18:45:31.383062 9 backup.go:69] Running br command with args: [backup full --pd=advanced-tidb-pd.aimo-tidb:2379 --storage=s3://tidb --s3.provider=minio --s3.endpoint=https://minio.local:9000]
I1118 18:45:31.413719 9 backup.go:93] [2022/11/18 18:45:31.413 +00:00] [INFO] [info.go:40] [“Welcome to Backup & Restore (BR)”] [release-version=v5.0.1] [git-hash=b3bd5495369244cfb1d3f2c49763a7fdb55d4ecd] [git-branch=heads/refs/tags/v5.0.1] [go-version=go1.13] [utc-build-time=“2021-04-23 06:02:57”] [race-enabled=false]
I1118 18:45:31.413750 9 backup.go:93] [2022/11/18 18:45:31.413 +00:00] [INFO] [common.go:471] [arguments] [__command=“br backup full”] [pd=“[advanced-tidb-pd.aimo-tidb:2379]”] [s3.endpoint=https://minio.local:9000] [s3.provider=minio] [storage=s3://tidb]
I1118 18:45:31.413916 9 backup.go:93] [2022/11/18 18:45:31.413 +00:00] [INFO] [client.go:214] [“[pd] create pd client with endpoints”] [pd-address=“[advanced-tidb-pd.aimo-tidb:2379]”]
I1118 18:45:31.421386 9 backup.go:93] [2022/11/18 18:45:31.421 +00:00] [INFO] [base_client.go:334] [“[pd] update member urls”] [old-urls=“[http://advanced-tidb-pd.aimo-tidb:2379]”] [new-urls=“[http://advanced-tidb-pd-0.advanced-tidb-pd-peer.aimo-tidb.svc:2379,http://advanced-tidb-pd-1.advanced-tidb-pd-peer.aimo-tidb.svc:2379,http://advanced-tidb-pd-2.advanced-tidb-pd-peer.aimo-tidb.svc:2379]”]
I1118 18:45:31.421403 9 backup.go:93] [2022/11/18 18:45:31.421 +00:00] [INFO] [base_client.go:346] [“[pd] switch leader”] [new-leader=http://advanced-tidb-pd-1.advanced-tidb-pd-peer.aimo-tidb.svc:2379] [old-leader=]
I1118 18:45:31.421444 9 backup.go:93] [2022/11/18 18:45:31.421 +00:00] [INFO] [base_client.go:126] [“[pd] init cluster id”] [cluster-id=6967171081824711717]
I1118 18:45:31.421455 9 backup.go:93] [2022/11/18 18:45:31.421 +00:00] [INFO] [client.go:238] [“[pd] create tso dispatcher”] [dc-location=global]
I1118 18:45:31.426520 9 backup.go:93] [2022/11/18 18:45:31.426 +00:00] [INFO] [client.go:214] [“[pd] create pd client with endpoints”] [pd-address=“[advanced-tidb-pd.aimo-tidb:2379]”]
I1118 18:45:31.432118 9 backup.go:93] [2022/11/18 18:45:31.432 +00:00] [INFO] [base_client.go:334] [“[pd] update member urls”] [old-urls=“[http://advanced-tidb-pd.aimo-tidb:2379]”] [new-urls=“[http://advanced-tidb-pd-0.advanced-tidb-pd-peer.aimo-tidb.svc:2379,http://advanced-tidb-pd-1.advanced-tidb-pd-peer.aimo-tidb.svc:2379,http://advanced-tidb-pd-2.advanced-tidb-pd-peer.aimo-tidb.svc:2379]”]
I1118 18:45:31.432133 9 backup.go:93] [2022/11/18 18:45:31.432 +00:00] [INFO] [base_client.go:346] [“[pd] switch leader”] [new-leader=http://advanced-tidb-pd-1.advanced-tidb-pd-peer.aimo-tidb.svc:2379] [old-leader=]
I1118 18:45:31.432169 9 backup.go:93] [2022/11/18 18:45:31.432 +00:00] [INFO] [base_client.go:126] [“[pd] init cluster id”] [cluster-id=6967171081824711717]
I1118 18:45:31.432178 9 backup.go:93] [2022/11/18 18:45:31.432 +00:00] [INFO] [client.go:238] [“[pd] create tso dispatcher”] [dc-location=global]
I1118 18:45:31.434658 9 backup.go:93] [2022/11/18 18:45:31.434 +00:00] [INFO] [conn.go:206] [“new mgr”] [pdAddrs=advanced-tidb-pd.aimo-tidb:2379]
I1118 18:45:31.435421 9 backup.go:93] [2022/11/18 18:45:31.435 +00:00] [INFO] [client.go:86] [“new backup client”]
I1118 18:45:31.481868 9 backup.go:93] [2022/11/18 18:45:31.481 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=1.971880233s]
I1118 18:45:33.458765 9 backup.go:93] [2022/11/18 18:45:33.458 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=2.46296939s]
I1118 18:45:35.925605 9 backup.go:93] [2022/11/18 18:45:35.925 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=6.357541376s]
I1118 18:45:42.287534 9 backup.go:93] [2022/11/18 18:45:42.287 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=11.785718824s]
I1118 18:45:54.079011 9 backup.go:93] [2022/11/18 18:45:54.078 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=21.93527864s]
I1118 18:46:16.021012 9 backup.go:93] [2022/11/18 18:46:16.020 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=47.833395104s]
I1118 18:47:03.859372 9 backup.go:93] [2022/11/18 18:47:03.859 +00:00] [WARN] [s3.go:686] [“failed to request s3, retrying”] [error=“RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority”] [backoff=1m29.900646144s]
I1118 18:48:33.765507 9 backup.go:93] [2022/11/18 18:48:33.765 +00:00] [INFO] [client.go:510] [“[pd] exit tso dispatcher”] [dc-location=global]
I1118 18:48:33.765977 9 backup.go:93] [2022/11/18 18:48:33.765 +00:00] [INFO] [client.go:510] [“[pd] exit tso dispatcher”] [dc-location=global]
I1118 18:48:33.765993 9 backup.go:93] [2022/11/18 18:48:33.765 +00:00] [INFO] [collector.go:191] [“Full backup failed summary”] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]
I1118 18:48:33.766003 9 backup.go:93] [2022/11/18 18:48:33.765 +00:00] [ERROR] [backup.go:41] [“failed to backup”] [error=“Bucket tidb is not accessible: RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority: [BR:ExternalStorage:ErrStorageInvalidConfig]invalid external storage config”] [errorVerbose=“[BR:ExternalStorage:ErrStorageInvalidConfig]invalid external storage config\nBucket tidb is not accessible: RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority\ngithub.com/pingcap/br/pkg/storage.newS3Storage\n\tgithub.com/pingcap/br@/pkg/storage/s3.go:283\ngithub.com/pingcap/br/pkg/storage.New\n\tgithub.com/pingcap/br@/pkg/storage/storage.go:145\ngithub.com/pingcap/br/pkg/storage.Create\n\tgithub.com/pingcap/br@/pkg/storage/storage.go:123\ngithub.com/pingcap/br/pkg/backup.(*Client).SetStorage\n\tgithub.com/pingcap/br@/pkg/backup/client.go:157\ngithub.com/pingcap/br/pkg/task.RunBackup\n\tgithub.com/pingcap/br@/pkg/task/backup.go:237\nmain.runBackupCommand\n\tgithub.com/pingcap/br@/cmd/br/backup.go:40\nmain.newFullBackupCommand.func1\n\tgithub.com/pingcap/br@/cmd/br/backup.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“main.runBackupCommand\n\tgithub.com/pingcap/br@/cmd/br/backup.go:41\nmain.newFullBackupCommand.func1\n\tgithub.com/pingcap/br@/cmd/br/backup.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203”]
I1118 18:48:33.766012 9 backup.go:93] Error: Bucket tidb is not accessible: RequestError: send request failed
I1118 18:48:33.766020 9 backup.go:93] caused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority: [BR:ExternalStorage:ErrStorageInvalidConfig]invalid external storage config
I1118 18:48:33.766079 9 backup.go:93] [2022/11/18 18:48:33.765 +00:00] [ERROR] [main.go:58] [“br failed”] [error=“Bucket tidb is not accessible: RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority: [BR:ExternalStorage:ErrStorageInvalidConfig]invalid external storage config”] [errorVerbose=“[BR:ExternalStorage:ErrStorageInvalidConfig]invalid external storage config\nBucket tidb is not accessible: RequestError: send request failed\ncaused by: Head https://minio.local:9000/tidb: x509: certificate signed by unknown authority\ngithub.com/pingcap/br/pkg/storage.newS3Storage\n\tgithub.com/pingcap/br@/pkg/storage/s3.go:283\ngithub.com/pingcap/br/pkg/storage.New\n\tgithub.com/pingcap/br@/pkg/storage/storage.go:145\ngithub.com/pingcap/br/pkg/storage.Create\n\tgithub.com/pingcap/br@/pkg/storage/storage.go:123\ngithub.com/pingcap/br/pkg/backup.(*Client).SetStorage\n\tgithub.com/pingcap/br@/pkg/backup/client.go:157\ngithub.com/pingcap/br/pkg/task.RunBackup\n\tgithub.com/pingcap/br@/pkg/task/backup.go:237\nmain.runBackupCommand\n\tgithub.com/pingcap/br@/cmd/br/backup.go:40\nmain.newFullBackupCommand.func1\n\tgithub.com/pingcap/br@/cmd/br/backup.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\tgith
ub.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“main.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:58\nruntime.main\n\truntime/proc.go:203”]
I1118 18:48:33.768475 9 backup.go:93]
E1118 18:48:33.768587 9 manager.go:292] backup cluster aimo-tid

2
| username: TiDBer_28PtrpL9 | Original post link

I’m confused and don’t know how to handle it. Does anyone know? Please let me know, thank you.
The MinIO cluster is working normally without any issues, using local certificate TLS.

3
| username: xfworld | Original post link

The authentication of this certificate did not pass, right? How about trying to switch to HTTP? Disable TLS.

4
| username: TiDBer_28PtrpL9 | Original post link

:sweat_smile: Trying to figure out how to trust it, don’t want to cancel MinIO’s HTTPS access. Tried adding the certificate to Linux trust, but it didn’t work. The certificate generation tool used is MinIO’s recommended certgen.

5
| username: xfworld | Original post link

Then you can try it more yourself. From a certain perspective, a self-signed certificate is not trusted.

6
| username: TiDBer_28PtrpL9 | Original post link

Still haven’t been able to solve it, can any experts give me some advice?

7
| username: TiDBer_28PtrpL9 | Original post link

In the end, I decided to cancel the HTTPS solution for MinIO. You can set up a standalone MinIO server instead.

8
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.