Error in Incremental Recovery of BR

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br增量恢复报错

| username: GreenGuan

Background:
Backup Phase: Full Backup + Incremental Backup
Restore Phase: Full Restore + Incremental Restore

Operation Process:
Full Backup Phase: Full backup using br version 4.0.1
Full Restore Phase: Full restore using br version 4.0.16
Incremental Backup Phase: Incremental backup using br version 4.0.1
There were some changes in partition tables in the intermediate database.
Incremental Restore Phase: Incremental restore using br version 4.0.16 (error), incremental restore using br version 4.0.1 (success)

Error as follows:

[2022/07/05 09:28:31.564 +08:00] [INFO] [info.go:40] ["Welcome to Backup & Restore (BR)"] [release-version=v4.0.16] [git-hash=e1b36e70163b32cda849ac0ddb47136f05c8de67] [git-branch=heads/refs/tags/v4.0.16] [go-version=go1.13] [utc-build-time="2021-12-02 04:16:27"] [race-enabled=false]
......
[2022/07/05 09:28:31.570 +08:00] [INFO] [base_client.go:252] ["[pd] switch leader"] [new-leader=http://xxxxx:3379] [old-leader=]
[2022/07/05 09:28:31.570 +08:00] [INFO] [base_client.go:102] ["[pd] init cluster id"] [cluster-id=7116436913537916717]
[2022/07/05 09:28:31.572 +08:00] [INFO] [conn.go:162] ["new mgr"] [pdAddrs=xxxxx:3379]
[2022/07/05 09:28:31.573 +08:00] [INFO] [tidb.go:72] ["new domain"] [store=tikv-7116436913537916717] ["ddl lease"=1s] ["stats lease"=-1ns]
[2022/07/05 09:28:31.577 +08:00] [INFO] [ddl.go:323] ["[ddl] start DDL"] [ID=75a7da49-f1bf-4e3d-9826-a4822f1cc8ba] [runWorker=true]
[2022/07/05 09:28:31.577 +08:00] [INFO] [manager.go:188] ["start campaign owner"] [ownerInfo="[ddl] /tidb/ddl/fg/owner"]
[2022/07/05 09:28:31.579 +08:00] [INFO] [ddl.go:312] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2022/07/05 09:28:31.579 +08:00] [INFO] [ddl_worker.go:131] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
[2022/07/05 09:28:31.579 +08:00] [INFO] [ddl_worker.go:131] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
[2022/07/05 09:28:31.627 +08:00] [INFO] [domain.go:148] ["full load InfoSchema success"] [usedSchemaVersion=0] [neededSchemaVersion=90] ["start time"=30.224528ms]
[2022/07/05 09:28:31.628 +08:00] [INFO] [domain.go:379] ["full load and reset schema validator"]
[2022/07/05 09:28:31.816 +08:00] [WARN] [version.go:238] ["cannot parse backup version"] [version=] [error=" is not in dotted-tri format"]
[2022/07/05 09:28:31.842 +08:00] [INFO] [client.go:186] ["load backupmeta"] [databases=2] [jobs=6]
[2022/07/05 09:28:31.846 +08:00] [WARN] [restore.go:546] ["set max-index-length to max(3072*4) to skip check index length in DDL"]
[2022/07/05 09:28:31.923 +08:00] [INFO] [session.go:2406] ["CRUCIAL OPERATION"] [conn=0] [schemaVersion=90] [cur_db=dbname] [sql="alter table xxxxx drop partition p20220627"] [user=]
[2022/07/05 09:28:31.933 +08:00] [INFO] [ddl_worker.go:262] ["[ddl] add DDL jobs"] ["batch count"=1] [jobs="ID:408, Type:drop partition, State:none, SchemaState:none, SchemaID:45, TableID:356, RowCount:0, ArgLen:1, start time: 2022-07-05 09:28:31.922 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0; "]
[2022/07/05 09:28:31.933 +08:00] [INFO] [ddl.go:537] ["[ddl] start DDL job"] [job="ID:408, Type:drop partition, State:none, SchemaState:none, SchemaID:45, TableID:356, RowCount:0, ArgLen:1, start time: 2022-07-05 09:28:31.922 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0"] [query="alter table xxxxx drop partition p20220627"]
[2022/07/05 09:28:32.443 +08:00] [INFO] [tidb.go:219] ["rollbackTxn for ddl/autocommit failed"]
[2022/07/05 09:28:32.470 +08:00] [WARN] [session.go:1384] ["run statement failed"] [schemaVersion=90] [error="[ddl:-1]json: cannot unmarshal array into Go value of type string"] [errorVerbose="[ddl:-1]json: cannot unmarshal array into Go value of type string\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\
github.com/pingcap/tidb/ddl.(*ddl).doDDLJob\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/ddl/ddl.go:578\
github.com/pingcap/tidb/ddl.(*ddl).DropTablePartition\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/ddl/ddl_api.go:2624\
github.com/pingcap/tidb/ddl.(*ddl).AlterTable\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/ddl/ddl_api.go:2107\
github.com/pingcap/tidb/executor.(*DDLExec).executeAlterTable\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/ddl.go:366\
github.com/pingcap/tidb/executor.(*DDLExec).Next\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/ddl.go:86\
github.com/pingcap/tidb/executor.Next\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/executor.go:262\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/adapter.go:531\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/adapter.go:413\
github.com/pingcap/tidb/executor.(*ExecStmt).Exec\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/adapter.go:366\
github.com/pingcap/tidb/session.runStmt\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/session/tidb.go:322\
github.com/pingcap/tidb/session.(*session).ExecuteStmt\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/session/session.go:1381\
github.com/pingcap/tidb/session.(*session).ExecuteInternal\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/session/session.go:1132\
github.com/pingcap/br/pkg/gluetidb.(*tidbSession).Execute\
\tgithub.com/pingcap/br@/pkg/gluetidb/glue.go:109\
github.com/pingcap/br/pkg/restore.(*DB).ExecDDL\
\tgithub.com/pingcap/br@/pkg/restore/db.go:79\
github.com/pingcap/br/pkg/restore.(*Client).ExecDDLs\
\tgithub.com/pingcap/br@/pkg/restore/client.go:500\
github.com/pingcap/br/pkg/task.RunRestore\
\tgithub.com/pingcap/br@/pkg/task/restore.go:320\
main.runRestoreCommand\
\tgithub.com/pingcap/br@/cmd/br/restore.go:25\
main.newFullRestoreCommand.func1\
\tgithub.com/pingcap/br@/cmd/br/restore.go:97\
github.com/spf13/cobra.(*Command).execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:842\
github.com/spf13/cobra.(*Command).ExecuteC\
\tgithub.com/spf13/cobra@v1.0.0/command.go:950\
github.com/spf13/cobra.(*Command).Execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:887\
main.main\
\tgithub.com/pingcap/br@/cmd/br/main.go:56\
runtime.main\
\truntime/proc.go:203\
runtime.goexit\
\truntime/asm_amd64.s:1357"] [session="{\
  \"currDBName\": \"dbname\",\
  \"id\": 0,\
  \"status\": 2,\
  \"strictMode\": false,\
  \"user\": null\
}"]
[2022/07/05 09:28:32.470 +08:00] [ERROR] [db.go:81] ["execute ddl query failed"] [query="alter table xxxxx drop partition p20220627"] [db=dbname] [historySchemaVersion=3710] [error="[ddl:-1]json: cannot unmarshal array into Go value of type string"] [errorVerbose="[ddl:-1]json: cannot unmarshal array into Go value of type string\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\
github.com/pingcap/tidb/ddl.(*ddl).doDDLJob\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/ddl/ddl.go:578\
github.com/pingcap/tidb/ddl.(*ddl).DropTablePartition\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/ddl/ddl_api.go:2624\
github.com/pingcap/tidb/ddl.(*ddl).AlterTable\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/ddl/ddl_api.go:2107\
github.com/pingcap/tidb/executor.(*DDLExec).executeAlterTable\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/ddl.go:366\
github.com/pingcap/tidb/executor.(*DDLExec).Next\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/ddl.go:86\
github.com/pingcap/tidb/executor.Next\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/executor.go:262\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/adapter.go:531\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/adapter.go:413\
github.com/pingcap/tidb/executor.(*ExecStmt).Exec\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/executor/adapter.go:366\
github.com/pingcap/tidb/session.runStmt\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/session/tidb.go:322\
github.com/pingcap/tidb/session.(*session).ExecuteStmt\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/session/session.go:1381\
github.com/pingcap/tidb/session.(*session).ExecuteInternal\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210714111333-67b641d5036c/session/session.go:1132\
github.com/pingcap/br/pkg/gluetidb.(*tidbSession).Execute\
\tgithub.com/pingcap/br@/pkg/gluetidb/glue.go:109\
github.com/pingcap/br/pkg/restore.(*DB).ExecDDL\
\tgithub.com/pingcap/br@/pkg/restore/db.go:79\
github.com/pingcap/br/pkg/restore.(*Client).ExecDDLs\
\tgithub.com/pingcap/br@/pkg/restore/client.go:500\
github.com/pingcap/br/pkg/task.RunRestore\
\tgithub.com/pingcap/br@/pkg/task/restore.go:320\
main.runRestoreCommand\
\tgithub.com/pingcap/br@/cmd/br/restore.go:25\
main.newFullRestoreCommand.func1\
\tgithub.com/pingcap/br@/cmd/br/restore.go:97\
github.com/spf13/cobra.(*Command).execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:842\
github.com/spf13/cobra.(*Command).ExecuteC\
\tgithub.com/spf13/cobra@v1.0.0/command.go:950\
github.com/spf13/cobra.(*Command).Execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:887\
main.main\
\tgithub.com/pingcap/br@/cmd/br/main.go:56\
runtime.main\
\truntime/proc.go:203\
runtime.goexit\
\truntime/asm_amd64.s:1357"] [stack="github.com/pingcap/br/pkg/restore.(*DB).ExecDDL\
\tgithub.com/pingcap/br@/pkg/restore/db.go:81\
github.com/pingcap/br/pkg/restore.(*Client).ExecDDLs\
\tgithub.com/pingcap/br@/pkg/restore/client.go:500\
github.com/pingcap/br/pkg/task.RunRestore\
\tgithub.com/pingcap/br@/pkg/task/restore.go:320\
main.runRestoreCommand\
\tgithub.com/pingcap/br@/cmd/br/restore.go:25\
main.newFullRestoreCommand.func1\
\tgithub.com/pingcap/br@/cmd/br/restore.go:97\
github.com/spf13/cobra.(*Command).execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:842\
github.com/spf13/cobra.(*Command).ExecuteC\
\tgithub.com/spf13/cobra@v1.0.0/command.go:950\
github.com/spf13/cobra.(*Command).Execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:887\
main.main\
\tgithub.com/pingcap/br@/cmd/br/main.go:56\
runtime.main\
\truntime/proc.go:203"]
[2022/07/05 09:28:32.470 +08:00] [INFO] [client.go:166] ["Restore client closed"]

Note: The br tool version 4.0.1 was used before the full restore phase, but

2
| username: TammyLi | Original post link

Currently, it is still recommended to use only full backup and full restore. As of the latest version 6.1.0, incremental backup and incremental restore in BR have not yet reached GA.

3
| username: xiaohetao | Original post link

What does your table structure look like?

The partition name p20220627 cannot be converted.

4
| username: xiaohetao | Original post link

It might be due to differences in partition table backups between versions.

5
| username: xiaohetao | Original post link

You can perform an incremental backup on version 4.0.16 and then conduct a recovery test to see if it succeeds.

6
| username: system | Original post link

This topic will be automatically closed 60 days after the last reply. No new replies are allowed.