Error Backing Up TiUniManager to AWS S3

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiUniManager备份到AWS S3出错

| username: 飞翔的企鹅

[TiDB Usage Environment] Production Environment / Testing / Poc

  • Production Environment

[TiDB Version]

  • v6.1.0

[Reproduction Path] What operations were performed to encounter the issue

  • The basic environment is all Amazon AWS EC2
  • Deployed TiDB v6.1.0 cluster using tiup (because I wanted to use TiUniManager, which only supports up to TiDB v6.1.0)
  • Deployed TiUniManager 1.0.2 on a separate EC2 according to the official documentation and successfully took over the cluster (all workflow tasks were successful).
  • Since the cluster is deployed on AWS, I also wanted to back up the data directly to AWS s3, so I created a Bucket on AWS and then used OpenAPI to modify the following configuration information (checked the configuration multiple times to ensure correctness):
    • BackupStorageType: s3 (default value unchanged)
    • BackupStoragePath: my-db-bak/rel
    • BackupS3AccessKey: xxxxx
    • BackupS3SecretAccessKey: xxxxx
    • BackupS3Endpoint: https://s3.ap-southeast-1.amazonaws.com
  • Set up a backup plan, went to the corresponding cluster → backup management → backup plan, and found that the backup failed after the scheduled time.
  • Went to the corresponding cluster and clicked manual backup, which also failed. The following is the log information.

[Encountered Issue: Problem Phenomenon and Impact]

  • TiUniManager workflow task failure screenshot:

  • Backup failed, key log information as follows:

{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin execute workflow BackupCluster, id fPZILNxURBWYmbBikcqM4w, node name backup","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin backupCluster","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"get cluster my-cluster tidb address from meta, [{IP:172.31.8.220 Port:4000} {IP:172.31.3.72 Port:4000}]","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"get cluster my-cluster user info from meta","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin do backup sql, request[{NodeID:kiHba6k-RYCf4Z5hCbqQug DbName: TableName: StorageAddress:s3://my-db-bak/rel/my-cluster/2023-11-01-13-33-56-full/?access-key=XXXXXXXXXXXXXXXXXXXX\\\u0026secret-access-key=aaaaaaaaaaaaaaaaaaaaaaaaaaa\\\u0026endpoint=https://s3.ap-southeast-1.amazonaws.com\\\u0026force-path-style=true DbConnParameter:{Username:EM_Backup_Restore Password:duw819273a IP:172.31.8.220 Port:4000} RateLimitM: Concurrency: CheckSum:}]","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"begin exec backup sql, request: {NodeID:kiHba6k-RYCf4Z5hCbqQug DbName: TableName: StorageAddress:s3://my-db-bak/rel/my-cluster/2023-11-01-13-33-56-full/?access-key=XXXXXXXXXXXXXXXXXXXX\\\u0026secret-access-key=aaaaaaaaaaaaaaaaaaaaaaaaaaa\\\u0026endpoint=https://s3.ap-southeast-1.amazonaws.com\\\u0026force-path-style=true DbConnParameter:{Username:EM_Backup_Restore Password:duw819273a IP:172.31.8.220 Port:4000} RateLimitM: Concurrency: CheckSum:}, bizId: kiHba6k-RYCf4Z5hCbqQug","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"error","msg":"query backup sql cmd failed Error 8124: Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request\n\tstatus code: 400, request id: S0VPEZTCA2M9P318, host id: GvO7VNT6L6Z/H7noQU7SwOaslyF+wpafwtavzgipLQIzBQBJ5xjZ+zuWFQIbgu11gfg0SfPO7OzTLsRhkVyoiw==","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"error","msg":"call backup api failed, Error 8124: Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request\n\tstatus code: 400, request id: S0VPEZTCA2M9P318, host id: GvO7VNT6L6Z/H7noQU7SwOaslyF+wpafwtavzgipLQIzBQBJ5xjZ+zuWFQIbgu11gfg0SfPO7OzTLsRhkVyoiw==","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"end backupCluster","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"workflow fPZILNxURBWYmbBikcqM4w of bizId my-cluster do node backup failed, Error 8124: Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request\n\tstatus code: 400, request id: S0VPEZTCA2M9P318, host id: GvO7VNT6L6Z/H7noQU7SwOaslyF+wpafwtavzgipLQIzBQBJ5xjZ+zuWFQIbgu11gfg0SfPO7OzTLsRhkVyoiw==","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"NsvRJ76rS6q5-wI0lHvhlg","level":"info","msg":"end execute workflow BackupCluster, id fPZILNxURBWYmbBikcqM4w, node name backup","time":"2023-11-01T13:34:00+08:00"}
{"Em-X-Trace-Id":"","level":"info","msg":"delete flow id fPZILNxURBWYmbBikcqM4w","time":"2023-11-01T13:34:00+08:00"}

[Resource Configuration]

  • TiDB * 2
  • PD * 3
  • TiKV * 3
  • Control * 1
  • Haproxy * 1

[Investigation Status]

Self-investigation, some parts are not understood, just started using TiDB for this project, purely personal guess, please be gentle…

First of all
I have successfully backed up to AWS s3 using br on the TiUniManager machine, indicating that the access-key and secret-access-key are correct and have the necessary permissions.

Attached is the TiUniManager source code repository address, which will be used later: TiUniManager

Question 1: Each parameter in the printed StorageAddress has an extra backslash

  • Similar to the following structure:
access-key=xxxxx\&secret-access-key=xxaaddww\&endpoint=xxaa\
  • The original log information is as follows:
StorageAddress:s3://my-db-bak/rel/my-cluster/2023-11-01-13-33-56-full/?access-key=XXXXXXXXXXXXXXXXXXXX\&secret-access-key=aaaaaaaaaaaaaaaaaaaaaaaaaaa\&endpoint=https://s3.ap-southeast-1.amazonaws.com\&force-path-style=true  
  • Checked the corresponding source code (/micro-cluster/cluster/backuprestore/executor.go line 314):

  • I don’t quite understand if concatenating this backslash has any impact.

Question 2: Is there an issue with the backup sql

  1. Checked the log error location and found that there was an error when executing the sql (db.QueryRow(brSQLCmd).Scan()), the corresponding source code location is /util/api/tidb/sql/backuprestore.go line 113:

  2. I wanted to print out the sql statement to check, but couldn’t print it, so I went to TiUniManager to check the logs and found the sql that was just executed, the content is as follows (the xxxx inside is not replaced by me, it was copied with it):

The sql is like this:

BACKUP DATABASE * TO 's3://my-db-bak/rel/my-cluster/2023-11-02-23-16-05-full/?access-key=xxxxxx&endpoint=https%3A%2F%2Fs3.ap-southeast-1.amazonaws.com&force-path-style=true&secret-access-key=xxxxxx'
  1. Newbie seeking advice: Can this sql be executed in mysql? I copied it to the command line and executed it, got the following error, same as the log error:
Error Code: 8124. Backup failed: error occurred when checking backupmeta file: BadRequest: Bad Request  status code: 400, request id: QHW82FH2KB55JRTV, host id: +iAXuNsdn7AxxdS5WDMj6OYhHiySwUlwrVat33DnV2RSZYYhyEGI6UwkR1YRoVXO9EBpih6AWPs=

If there is really no way, I can only take another route:

  1. Do not back up to s3, try using nfs;
  2. Abandon TiUniManager backup, use br instead;
  3. Build the source code myself and then find a way;

Seeking advice from community experts, is there any solution? Is my usage incorrect or does TiUniManager not support backup to s3?

I’ve been struggling for two days, too tangled, hope the experts can help take a look :sob: :sob: :sob:

| username: xfworld | Original post link

Try using BR directly.

Refer to this:

Haven’t paid much attention to TiUniManager… :rofl:

| username: tidb菜鸟一只 | Original post link

I’ve never used TiUniManager…

| username: onlyacat | Original post link

The last update to the tiunimanager repository was in April this year, with only 4 commits this year. I’m not sure if it’s still being maintained.

If you need a management platform, you might want to check out the commercial version of TiDB, which includes TEM.

| username: 飞翔的企鹅 | Original post link

:joy: :joy: Thanks for the reply. I downgraded the version from 7.1.2 to 6.1.0 just to use TiUniManager, but I didn’t expect this… Logically, the officially released tools shouldn’t have issues. I’ve been researching for two days, it’s so frustrating :sweat_smile: :sweat_smile:

| username: 飞翔的企鹅 | Original post link

Thank you for the reply. I would like to ask, if we don’t use TiUniManager, what do we use for cluster monitoring?

| username: xfworld | Original post link

Cluster monitoring can be done using dashboard and Grafana.

| username: 飞翔的企鹅 | Original post link

Thank you for the reply :pray: :pray:. It’s unlikely that the official release would be a half-finished product… I saw that the documentation introduces this tool in great detail. If that’s the case, I would be really upset, having downgraded the version specifically :sob: :sob:. By the way, do you have a link to the introduction of TEM? I couldn’t find any relevant information just now.

| username: 飞翔的企鹅 | Original post link

I’ll bump this up myself. Doesn’t anyone have a need to use TiUniManager? :sob: :sob: :sob:

| username: xfworld | Original post link

The commercial version might better meet your expectations, tiem.

| username: 飞翔的企鹅 | Original post link

I didn’t see any introduction of TIEM in the commercial version either. Everything is deployed, just missing a backup :sob: :sob:

| username: xfworld | Original post link

Check this out


The features of the community edition are introduced in the link above.