Backup Data to KS3 Storage Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br备份数据到ks3存储报错

| username: liyuntang

【TiDB Usage Environment】Production Environment
【TiDB Version】
TiDB version is v6.5.0, deployed in Kingsoft Cloud’s KCE

【Reproduction Path】
Backup the entire cluster data to KS3 storage using the BE tool configuration

【Encountered Issues: Problem Phenomenon and Impact】

  1. The BR tool versions v5.4.1, v6.5.0, and v6.5.1 cannot back up the TiDB cluster of version 6.5.0 to KS3.
  2. Tried both tidb-operator versions 1.3 and 1.4, all reported errors.

【Resource Configuration】

var version string
var ns string

func init() {
	flag.StringVar(&ns, "ns", "eeae42ce-c439-442b-863b-0119929a076c", "name space")
	flag.StringVar(&version, "v", "v5.4.1", "br version,support v6.5.1 and v5.4.1 default v5.4.1")
}

func newConfig() *v1alpha1.Backup {
	image := fmt.Sprintf("hub.kce.ksyun.com/nosql/tidb/br:%s", version)
	sendCredToTikv := true
	storageClassName := "openebs-hostpath"
	suffix := time.Now().Unix()
	return &v1alpha1.Backup{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Backup",
			APIVersion: "pingcap.com/v1alpha1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name: fmt.Sprintf("liyuntang-%d", suffix),
		},
		Spec: v1alpha1.BackupSpec{
			//ServiceAccount: "tidb-backup-manager",
			BR: &v1alpha1.BRConfig{
				Cluster:          "basic",
				ClusterNamespace: ns,
				SendCredToTikv:   &sendCredToTikv,
			},
			ToolImage:        image,
			ImagePullSecrets: []corev1.LocalObjectReference{{Name: "nosql-image"}},
			StorageProvider: v1alpha1.StorageProvider{
				S3: &v1alpha1.S3StorageProvider{
					Provider:   "other",
					Region:     "BEIJING",
					Bucket:     "tidb-cket-bj",
					Endpoint:   "http://kal.ksyun.com",
					SecretName: "kt",
					Prefix:     fmt.Sprintf("200074/%s/manual/%d", ns, suffix),
				},
			},
			StorageClassName: &storageClassName,
			Type:             v1alpha1.BackupTypeFull,
			CleanPolicy:      v1alpha1.CleanPolicyTypeDelete,
		},
	}
}

【Attachment: Screenshot/Log/Monitoring】

[root@vm10-32-1-164 tmp]# kubectl logs backup-2-1-liyuntang-1679477820-qvc24 -n $ns
Create rclone.conf file.
/tidb-backup-manager backup --namespace=eeae42ce-c439-442b-863b-0119929a076c --backupName=2-1-liyuntang-1679477820 --tikvVersion=v6.5.0
I0322 17:37:11.709234       9 backup.go:72] start to process backup eeae42ce-c439-442b-863b-0119929a076c/2-1-liyuntang-1679477820
I0322 17:37:11.725730       9 backup_status_updater.go:86] Backup: [eeae42ce-c439-442b-863b-0119929a076c/2-1-liyuntang-1679477820] updated successfully
E0322 17:37:11.730469       9 backup_status_updater.go:89] Failed to update backup [eeae42ce-c439-442b-863b-0119929a076c/2-1-liyuntang-1679477820], error: Operation cannot be fulfilled on backups.pingcap.com "2-1-liyuntang-1679477820": the object has been modified; please apply your changes to the latest version and try again
I0322 17:37:11.749029       9 backup_status_updater.go:86] Backup: [eeae42ce-c439-442b-863b-0119929a076c/2-1-liyuntang-1679477820] updated successfully
I0322 17:37:11.749097       9 backup.go:69] Running br command with args: [backup full --pd=basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379 --storage=s3://tidb-backup-bucket-bj/2000102274/eeae42ce-c439-442b-863b-0119929a076c/manual/2-1-liyuntang-1679477820 --s3.region=BEIJING --s3.provider=other --s3.endpoint=http://ks3-cn-beijing-internal.ksyun.com]
I0322 17:37:11.839176       9 backup.go:93] [2023/03/22 17:37:11.838 +08:00] [INFO] [info.go:49] ["Welcome to Backup & Restore (BR)"] [release-version=v6.5.0] [git-hash=706c3fa3c526cdba5b3e9f066b1a568fb96c56e3] [git-branch=heads/refs/tags/v6.5.0] [go-version=go1.19.3] [utc-build-time="2022-12-27 03:41:53"] [race-enabled=false]
I0322 17:37:11.839245       9 backup.go:93] [2023/03/22 17:37:11.839 +08:00] [INFO] [common.go:718] [arguments] [__command="br backup full"] [pd="[basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379]"] [s3.endpoint=http://ks3-cn-beijing-internal.ksyun.com] [s3.provider=other] [s3.region=BEIJING] [storage=s3://tidb-backup-bucket-bj/2000102274/eeae42ce-c439-442b-863b-0119929a076c/manual/2-1-liyuntang-1679477820]
I0322 17:37:11.839311       9 backup.go:93] [2023/03/22 17:37:11.839 +08:00] [INFO] [backup.go:188] ["use checkpoint's default GC TTL"] ["GC TTL"=4320]
I0322 17:37:11.839465       9 backup.go:93] [2023/03/22 17:37:11.839 +08:00] [INFO] [conn.go:147] ["new mgr"] [pdAddrs=basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379]
I0322 17:37:11.855475       9 backup.go:93] [2023/03/22 17:37:11.855 +08:00] [INFO] [client.go:405] ["[pd] create pd client with endpoints"] [pd-address="[basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379]"]
I0322 17:37:11.873635       9 backup.go:93] [2023/03/22 17:37:11.873 +08:00] [INFO] [base_client.go:360] ["[pd] update member urls"] [old-urls="[http://basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379]"] [new-urls="[http://basic-pd-0.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379,http://basic-pd-1.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379,http://basic-pd-2.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379]"]
I0322 17:37:11.874295       9 backup.go:93] [2023/03/22 17:37:11.873 +08:00] [INFO] [base_client.go:378] ["[pd] switch leader"] [new-leader=http://basic-pd-0.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379] [old-leader=]
I0322 17:37:11.874353       9 backup.go:93] [2023/03/22 17:37:11.873 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7205466925320758990]
I0322 17:37:11.874368       9 backup.go:93] [2023/03/22 17:37:11.873 +08:00] [INFO] [client.go:698] ["[pd] tso dispatcher created"] [dc-location=global]
I0322 17:37:11.893793       9 backup.go:93] [2023/03/22 17:37:11.893 +08:00] [INFO] [conn.go:121] ["checked alive KV stores"] [aliveStores=4] [totalStores=4]
I0322 17:37:11.893830       9 backup.go:93] [2023/03/22 17:37:11.893 +08:00] [INFO] [client.go:405] ["[pd] create pd client with endpoints"] [pd-address="[basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379]"]
I0322 17:37:11.909555       9 backup.go:93] [2023/03/22 17:37:11.909 +08:00] [INFO] [base_client.go:360] ["[pd] update member urls"] [old-urls="[http://basic-pd.eeae42ce-c439-442b-863b-0119929a076c:2379]"] [new-urls="[http://basic-pd-0.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379,http://basic-pd-1.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379,http://basic-pd-2.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379]"]
I0322 17:37:11.909707       9 backup.go:93] [2023/03/22 17:37:11.909 +08:00] [INFO] [base_client.go:378] ["[pd] switch leader"] [new-leader=http://basic-pd-0.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379] [old-leader=]
I0322 17:37:11.909726       9 backup.go:93] [2023/03/22 17:37:11.909 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7205466925320758990]
I0322 17:37:11.909796       9 backup.go:93] [2023/03/22 17:37:11.909 +08:00] [INFO] [client.go:698] ["[pd] tso dispatcher created"] [dc-location=global]
I0322 17:37:11.926315       9 backup.go:93] [2023/03/22 17:37:11.926 +08:00] [INFO] [tidb.go:77] ["new domain"] [store=tikv-7205466925320758990] ["ddl lease"=1s] ["stats lease"=-1ns] ["index usage sync lease"=0s]
I0322 17:37:11.959845       9 backup.go:93] [2023/03/22 17:37:11.959 +08:00] [INFO] [domain.go:2280] [acquireServerID] [serverID=1157728] ["lease id"=250787077b919dfd]
I0322 17:37:11.966885       9 backup.go:93] [2023/03/22 17:37:11.966 +08:00] [WARN] [info.go:245] ["init TiFlashReplicaManager"] ["pd addrs"="[basic-pd-1.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379,basic-pd-0.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379,basic-pd-2.basic-pd-peer.eeae42ce-c439-442b-863b-0119929a076c.svc:2379]"]
I0322 17:37:12.131522       9 backup.go:93] [2023/03/22 17:37:12.131 +08:00] [INFO] [domain.go:220] ["full load InfoSchema success"] [currentSchemaVersion=0] [neededSchemaVersion=138] ["start time"=118.796307ms]
I0322 17:37:12.135279       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [domain.go:487] ["full load and reset schema validator"]
I0322 17:37:12.135340       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [ddl.go:701] ["[ddl] start DDL"] [ID=97618410-f614-4975-a3b4-512021ba6fd9] [runWorker=false]
I0322 17:37:12.135385       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [ddl.go:647] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
I0322 17:37:12.135514       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [WARN] [env.go:52] ["[ddl-ingest] initialize environment failed"] ["storage limitation"="only support TiKV storage"] ["current storage"=unistore] ["lightning is initialized"=false]
I0322 17:37:12.135618       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [ddl_worker.go:171] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
I0322 17:37:12.135702       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [ddl_worker.go:171] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
I0322 17:37:12.135956       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [env.go:108] ["[ddl-ingest] the lightning sorted dir"] ["data path:"=/tmp/tidb/tmp_ddl-4000]
I0322 17:37:12.136048       9 backup.go:93] [2023/03/22 17:37:12.135 +08:00] [INFO] [env.go:75] ["[ddl-ingest] init global lightning backend environment finished"] ["memory limitation"=2147483648] ["sort path disk quota"=107374182400] ["max open file number"=1048576] ["lightning is initialized"=true]
I0322 17:37:12.136125       9 backup.go:93] [2023/03/22 17:37:12.136 +08:00] [INFO] [owner_daemon.go:70] ["begin advancer daemon"] [daemon-id=LogBackup::Advancer]
I0322 17:37:12.136144       9 backup.go:93] [2023/03/22 17:37:12.136 +08:00] [INFO] [manager.go:151] ["start campaign owner"] [ownerInfo="[log-backup] /tidb/br-stream/owner"]
I0322 17:37:12.139580       9 backup.go:93] [2023/03/22 17:37:12.139 +08:00] [INFO] [owner_daemon.go:77] ["begin running daemon"] [id=01614d1c-3150-457f-b756-1279fe339fe3] [daemon-id=LogBackup::Advancer]
I0322 17:37:12.152950       9 backup.go:93] [2023/03/22 17:37:12.152 +08:00] [INFO] [backup.go:369] ["get new_collations_enabled_on_first_bootstrap config from system table"] [new_collation_enabled=True]
I0322 17:37:13
| username: Billmay表妹 | Original post link

You can take a look.

| username: liyuntang | Original post link

The left side is version 5.4.1, and the right side is version 6.5.0.

| username: shonge | Original post link

Change the s3 provider to aws and try:

StorageProvider: v1alpha1.StorageProvider{
    S3: &v1alpha1.S3StorageProvider{
	Provider:   "aws"
| username: liyuntang | Original post link

Still reporting an error

| username: tidb菜鸟一只 | Original post link

Is there an issue with the region you specified for your S3?

| username: dba-kit | Original post link

Try adding force-path-style=false to see if it works.

| username: Jellybean | Original post link

Yes, looking at the error log, it seems that the parameters in the startup command are incorrect.

Make sure to check the relevant parameters for the bucket and region.

| username: liyuntang | Original post link

I am using KS3, not S3, so the region configuration should not be an issue. The format provided by the official documentation is like this:

| username: liyuntang | Original post link

The parameters are fine.

| username: liyuntang | Original post link

This is the result of running br using the command line. From the logs, it can be seen that br directly accesses AWS’s S3 without accessing the specified endpoint. Therefore, it can be largely determined that br handles the scenario where provider=other accordingly.

| username: shonge | Original post link

It looks like this issue was introduced by this PR:

GetBucketRegionWithClient only supports AWS S3:

| username: liyuntang | Original post link

Is there any plan to support other S3-compatible storage? If so, when is it expected to be released?

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.