Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: TiUniManager备份到AWS S3出错

[TiDB Usage Environment] Production Environment / Testing / Poc
- Production Environment
[TiDB Version]
- v6.1.0
[Reproduction Path] What operations were performed to encounter the issue
- The basic environment is all Amazon
AWS EC2
- Deployed
TiDB v6.1.0
cluster usingtiup
(because I wanted to useTiUniManager
, which only supports up toTiDB v6.1.0
) - Deployed
TiUniManager 1.0.2
on a separateEC2
according to the official documentation and successfully took over the cluster (all workflow tasks were successful). - Since the cluster is deployed on
AWS
, I also wanted to back up the data directly toAWS s3
, so I created aBucket
onAWS
and then usedOpenAPI
to modify the following configuration information (checked the configuration multiple times to ensure correctness):- BackupStorageType: s3 (default value unchanged)
- BackupStoragePath: my-db-bak/rel
- BackupS3AccessKey: xxxxx
- BackupS3SecretAccessKey: xxxxx
- BackupS3Endpoint: https://s3.ap-southeast-1.amazonaws.com
- Set up a backup plan, went to the corresponding cluster → backup management → backup plan, and found that the backup failed after the scheduled time.
- Went to the corresponding cluster and clicked manual backup, which also failed. The following is the log information.
[Encountered Issue: Problem Phenomenon and Impact]
-
TiUniManager
workflow task failure screenshot:
-
Backup failed, key log information as follows:
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin execute workflow BackupCluster, id fPZILNxURBWYmbBikcqM4w, node name backup","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin backupCluster","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"get cluster my-cluster tidb address from meta, [{IP:172.31.8.220 Port:4000} {IP:172.31.3.72 Port:4000}]","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"get cluster my-cluster user info from meta","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin do backup sql, request[{NodeID:kiHba6k-RYCf4Z5hCbqQug DbName: TableName: StorageAddress:s3://my-db-bak/rel/my-cluster/2023-11-01-13-33-56-full/?access-key=XXXXXXXXXXXXXXXXXXXX\\\u0026secret-access-key=aaaaaaaaaaaaaaaaaaaaaaaaaaa\\\u0026endpoint=https://s3.ap-southeast-1.amazonaws.com\\\u0026force-path-style=true DbConnParameter:{Username:EM_Backup_Restore Password:duw819273a IP:172.31.8.220 Port:4000} RateLimitM: Concurrency: CheckSum:}]","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin exec backup sql, request: {NodeID:kiHba6k-RYCf4Z5hCbqQug DbName: TableName: StorageAddress:s3://my-db-bak/rel/my-cluster/2023-11-01-13-33-56-full/?access-key=XXXXXXXXXXXXXXXXXXXX\\\u0026secret-access-key=aaaaaaaaaaaaaaaaaaaaaaaaaaa\\\u0026endpoint=https://s3.ap-southeast-1.amazonaws.com\\\u0026force-path-style=true DbConnParameter:{Username:EM_Backup_Restore Password:duw819273a IP:172.31.8.220 Port:4000} RateLimitM: Concurrency: CheckSum:}, bizId: kiHba6k-RYCf4Z5hCbqQug","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"error","msg":"query backup sql cmd failed Error 8124: Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request\n\tstatus code: 400, request id: S0VPEZTCA2M9P318, host id: GvO7VNT6L6Z/H7noQU7SwOaslyF+wpafwtavzgipLQIzBQBJ5xjZ+zuWFQIbgu11gfg0SfPO7OzTLsRhkVyoiw==","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"error","msg":"call backup api failed, Error 8124: Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request\n\tstatus code: 400, request id: S0VPEZTCA2M9P318, host id: GvO7VNT6L6Z/H7noQU7SwOaslyF+wpafwtavzgipLQIzBQBJ5xjZ+zuWFQIbgu11gfg0SfPO7OzTLsRhkVyoiw==","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"end backupCluster","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"workflow fPZILNxURBWYmbBikcqM4w of bizId my-cluster do node backup failed, Error 8124: Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request\n\tstatus code: 400, request id: S0VPEZTCA2M9P318, host id: GvO7VNT6L6Z/H7noQU7SwOaslyF+wpafwtavzgipLQIzBQBJ5xjZ+zuWFQIbgu11gfg0SfPO7OzTLsRhkVyoiw==","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"end execute workflow BackupCluster, id fPZILNxURBWYmbBikcqM4w, node name backup","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"","level":"info","msg":"delete flow id fPZILNxURBWYmbBikcqM4w","time":"2023-11-01T13:34:00+08:00"}
[Resource Configuration]
- TiDB * 2
- PD * 3
- TiKV * 3
- Control * 1
- Haproxy * 1
[Investigation Status]
Self-investigation, some parts are not understood, just started using TiDB for this project, purely personal guess, please be gentle…
First of all
I have successfully backed up to AWS s3
using br
on the TiUniManager
machine, indicating that the access-key
and secret-access-key
are correct and have the necessary permissions.
Attached is the TiUniManager
source code repository address, which will be used later: TiUniManager
Question 1: Each parameter in the printed StorageAddress
has an extra backslash
- Similar to the following structure:
access-key=xxxxx\&secret-access-key=xxaaddww\&endpoint=xxaa\
- The original log information is as follows:
StorageAddress:s3://my-db-bak/rel/my-cluster/2023-11-01-13-33-56-full/?access-key=XXXXXXXXXXXXXXXXXXXX\&secret-access-key=aaaaaaaaaaaaaaaaaaaaaaaaaaa\&endpoint=https://s3.ap-southeast-1.amazonaws.com\&force-path-style=true
-
Checked the corresponding source code (
/micro-cluster/cluster/backuprestore/executor.go
line 314):
-
I don’t quite understand if concatenating this backslash has any impact.
Question 2: Is there an issue with the backup sql
-
Checked the log error location and found that there was an error when executing the
sql
(db.QueryRow(brSQLCmd).Scan()
), the corresponding source code location is/util/api/tidb/sql/backuprestore.go
line 113:
-
I wanted to print out the
sql
statement to check, but couldn’t print it, so I went toTiUniManager
to check the logs and found thesql
that was just executed, the content is as follows (thexxxx
inside is not replaced by me, it was copied with it):
The sql is like this:
BACKUP DATABASE * TO 's3://my-db-bak/rel/my-cluster/2023-11-02-23-16-05-full/?access-key=xxxxxx&endpoint=https%3A%2F%2Fs3.ap-southeast-1.amazonaws.com&force-path-style=true&secret-access-key=xxxxxx'
- Newbie seeking advice: Can this
sql
be executed inmysql
? I copied it to the command line and executed it, got the following error, same as the log error:
Error Code: 8124. Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request status code: 400, request id: QHW82FH2KB55JRTV, host id: +iAXuNsdn7AxxdS5WDMj6OYhHiySwUlwrVat33DnV2RSZYYhyEGI6UwkR1YRoVXO9EBpih6AWPs=
If there is really no way, I can only take another route:
- Do not back up to
s3
, try usingnfs
; - Abandon
TiUniManager
backup, usebr
instead; - Build the source code myself and then find a way;
Seeking advice from community experts, is there any solution? Is my usage incorrect or does TiUniManager not support backup to s3?
I’ve been struggling for two days, too tangled, hope the experts can help take a look