BackupSchedule fails to run

TiDB Ops & Admin
1

Hey TiDB folks,

Can I get help debugging a BackupSchedule that fails to run?

To create the BackupSchedule, I first started with a Backup config that works on my cluster.

Then I created a BackupSchedule and copy-pasted my config into the backupTemplate and logBackupTemplate fields, though I changed the destination bucket for the logBackupTemplate.
Here it is.
IIUC, this should work, because those fields use the Backup CRD.

When I apply this, I get these errors from the controller manager.

Any idea what’s going on?
It seems to maybe have something to do with storage.

Thanks!

2

What versions do the tidb-operator and tidb-cluster use? Do you upgrade them recently?

3

Sorry for the delay - I was traveling.

I created the CRDs using this command, run a couple days ago:

kubectl create -f \
  https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/crd.yaml

My ClusterConfig specifies version 6.5.0.

Does that tell you what you need to know?

EDIT: I just noticed I’m using master and not a release branch. Let me try it with v1.4.4 and get back to you.

4

I just deleted and recreated the cluster and setup TiDB with these commands:

kubectl create -f \
  https://raw.githubusercontent.com/pingcap/tidb-operator/v1.4.4/manifests/crd.yaml

helm install \
  --namespace tidb-admin --create-namespace \
  tidb-operator pingcap/tidb-operator \
  --version v1.4.4

wget \
  https://raw.githubusercontent.com/pingcap/tidb-operator/v1.4.4/manifests/backup/backup-rbac.yaml \
  -O k8s/backup-rbac.yaml
kubectl apply -f k8s/backup-rbac.yaml -n tidb-cluster

kubectl -n tidb-cluster apply -f k8s/tidb-cluster.yaml

Here is tidb-cluster.yaml.

The errors are the same as before.
Here’s a fresh sample.

5

A small bug here, pls add kv pair “backupMode: log” to “logBackupTemplate” as a workaround.

6

Thank you!

That works, but it’s not deleting old backups.

Here’s the backup folder:

Screenshot 2023-03-24 at 10.57.04 PM
Screenshot 2023-03-24 at 10.57.04 PM2684×672 156 KB

Here’s the log backup folder:

Screenshot 2023-03-24 at 10.57.13 PM
Screenshot 2023-03-24 at 10.57.13 PM2670×162 17.3 KB

7

pls set backupTemplate.spec.cleanPolicy to “Delete”

8

That didn’t work - old backups still aren’t being deleted.
Also, if I’m reading this correctly cleanPolicy only applies when the CR is deleted.

Any more ideas?

(Sorry for the long lag, I’ve been traveling.)

9

set backupTemplate.spec.cleanPolicy to “Delete” only works for new backup, for the old one, you have to do manual cleanup