BR backup successful in cluster A, but restore failed in cluster B

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: BR在a集群备份成功，用在b集群还原失败

| username: Steve阿辉

【TiDB Usage Environment】Production Environment
【TiDB Version】6.1.2
【Reproduction Path】Operations that led to the issue: Cluster A was upgraded from version 5.3.1 to 6.1.2, and the BR tool was also upgraded. Additionally, we created a new Cluster B.
【Encountered Issue: Symptoms and Impact】After performing the operations, we found that the historical BR backup from version 5.3.1 could be successfully restored in Cluster B (version 6.1.2), but data queries returned no results. When performing a BR backup of the current data from Cluster A and restoring it in Cluster B, the restoration failed. We are now seeking a reliable data synchronization component solution to synchronize data from Cluster A to Cluster B, after which the business will be migrated to Cluster B.
【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】Both clusters are using BR version 6.1.2. Can we use a different version for the operation? How can we troubleshoot the specific issue?

(base) [root@tidb0003 cluster]# ./br backup table \

–db spider --table aba_quarterly
–pd “172.16.16.13:2379”
-s “local:///home/br-backup/table/quarterly/20221221”
–log-file backupfull.log

Detail BR log in backupfull.log

Table Backup <--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%

Checksum <------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%

[2022/12/22 04:45:15.860 +08:00] [INFO] [collector.go:69] [“Table Backup success summary”] [total-ranges=801] [ranges-succeed=801] [ranges-failed=0] [backup-checksum=1m10.426031647s] [backup-fast-checksum=578.474µs] [backup-total-ranges=8] [backup-total-regions=557] [total-take=6m40.065563235s] [Size=15248693413] [BackupTS=438214358710616065] [total-kv=269627120] [total-kv-size=41.97GB] [average-speed=104.9MB/s] [backup-data-size(after-compressed)=15.25GB]

(base) [root@tidb0003 cluster]#

– Restore failed

[tidb@tiup-new tools]$ ./br restore table \

–db spider --table aba_quarterly
–pd “172.16.16.67:2379”
-s “local:///home/br-backup/table/quarterly/20221221”
–log-file backupfull.log
–check-requirements=false

Detail BR log in backupfull.log

[2022/12/22 04:49:49.665 +08:00] [INFO] [collector.go:69] [“Table Restore failed summary”] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]

Error: the config ‘new_collations_enabled_on_first_bootstrap’ not match, upstream:False, downstream: True: [BR:Common:ErrUnknown]internal error

[tidb@tiup-new tools]$

2
| username: wakaka | Original post link

It looks like the initialization parameters on both sides are inconsistent. This affects the database table character set, and it is possible that the new cluster does not support it because this parameter is not enabled.

3
| username: wakaka | Original post link

Additionally, if the data volume is not large, you can consider using Dumpling and Lightning for backup and recovery. BR is a physical backup, while Dumpling is a logical backup.

4
| username: Steve阿辉 | Original post link

We added –check-requirements=false at the end to solve the true and false issue. Regarding the character set you mentioned, how can we check what character set the database is using? Because when we use Navicat to check, the character set is: utf8mb4, and the collation is utf8mb4_bin, both sides are the same.

5
| username: 裤衩儿飞上天 | Original post link

The image you provided is not visible. Please provide the text you need translated.

6
| username: 裤衩儿飞上天 | Original post link

My understanding is that you need to first set this parameter consistently upstream and downstream, and then skip this configuration check in order to restore it properly.

7
| username: 裤衩儿飞上天 | Original post link

First, check whether the values of this configuration item are the same upstream and downstream.

8
| username: hey-hoho | Original post link

To summarize, your cluster A was upgraded from 5.3.1 to 6.1.2, and cluster B is a newly created 6.1.2.

The parameter new_collations_enabled_on_first_bootstrap controls whether TiDB supports new collations, such as utf8mb4_general_ci. Before version 6.0, the default was false, and after 6.0, the default is true. This parameter can only be set during cluster deployment. If you didn’t manually change this parameter during the deployment of clusters A and B, this would cause the parameter values to be inconsistent between your clusters A and B.

BR (Backup & Restore) is a physical backup and restore tool that requires the new_collations_enabled_on_first_bootstrap parameter to be consistent between the two clusters. This is unrelated to the character set collation used by your database. Refer to the BR documentation for usage restrictions:

image
image1380×762 113 KB

Conclusion: Backup data from cluster A cannot be restored in cluster B. The recommended approach is to export from cluster A using Dumpling and import into cluster B using Lightning.

9
| username: 裤衩儿飞上天 | Original post link

I have a cluster that was upgraded from 5.4.1 to 6.1.2. Why is the value of new_collations_enabled_on_first_bootstrap set to true? I didn’t change this parameter during the upgrade. Before version 6.0, it should have been false.

TiDB root@10.18.13.224:test>
TiDB root@10.18.13.224:test> select version();
+--------------------+
| version()          |
+--------------------+
| 5.7.25-TiDB-v6.1.2 |
+--------------------+
1 row in set
Time: 0.017s
TiDB root@10.18.13.224:test> show config where name like '%new_coll%'
+------+-------------------+-------------------------------------------+-------+
| Type | Instance          | Name                                      | Value |
+------+-------------------+-------------------------------------------+-------+
| tidb | 10.18.13.227:4000 | new_collations_enabled_on_first_bootstrap | true  |
| tidb | 10.18.13.226:4000 | new_collations_enabled_on_first_bootstrap | true  |
| tidb | 10.18.13.225:4000 | new_collations_enabled_on_first_bootstrap | true  |
+------+-------------------+-------------------------------------------+-------+
3 rows in set
Time: 0.061s
TiDB root@10.18.13.224:test>
10
| username: 会飞的土拨鼠 | Original post link

It is recommended to keep the versions of the br tool consistent for import and export. If the device has enough storage space, you can also use dumpling and lightling for backup and recovery. Use version 6.1.2 tools for TiDB v6.1.2 and version 5.3.1 tools for TiDB v5.3.1.

11
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.