As mentioned, I am working on a complete backup and log backup solution for TiDB.
After using the tiup br log start command to specify a shared directory to start log backup, I can see various files generated in the directory.
I want to know if it is possible to implement an operation,
such as once a day, packaging the log backup files generated that day into a backup set for storage, and then clearing the local log files.
Then, during recovery, first restore the full backup, then merge all the packaged backup sets into one directory, and perform the PIRT recovery operation.
If possible, how should it be done, and how to merge the packaged files?
Because the directory seems to contain a bunch of SST files, which can be understood as the original information of the region, and it includes all the MVCC information from the start of the backup to now to achieve PITR. This is probably not something that file merging can solve.
Because some SST files may not be fully written, restoring SST files by day will definitely have overlaps, and if the order of operations by day is wrong, the restored logs will be unusable.
If you have to upload once a day, I feel it’s safer to stop the backup with tiup br log stop every day, then back it up, restart the task according to the timestamp, and change the saved path. This might work.
I haven’t tried it either. You can give it a try.
It’s best to use S3, which is very hassle-free. The reason you’re going through all this trouble is because you’re using a shared directory.
Hello, I have a question to confirm. If we perform a full backup plus log backup using a shared directory, and since the log backup is done once a day, during restoration, we first restore the full backup. However, there are multiple incremental backups. Do we need to restore multiple times or can it be done in one go?
It’s hard to answer without actually trying it. Personally, I feel that restoring in this way is very difficult and unreliable. Let’s see if any other experts have good practices.
I tried it yesterday. You can stop the log backup first, then copy all the logs and package them.
When restoring, you can start from the full backup set and restore one by one, as long as the ts is continuous.
However, the recoverable time will have a few minutes difference from the backup time. I wonder if there is a forced checkpoint operation, and whether it is possible to force log generation to reduce the time difference.
Are you saying that during recovery, you first perform a full restore, and then restore the incremental data in batches according to the chronological order? For example, if the full backup is on 2024-05-01, then you have incremental backups on the 2nd, 3rd, 4th, and so on. During recovery, you first restore all data up to the 1st, and then perform the restore for the 2nd, 3rd, and so on in sequence. Is that correct?
However, you can use the br log metadata command to ensure that the most recent recoverable time point includes the previous full day. But if you want to precisely control it down to the second level, it might indeed be a bit difficult.
If that’s the case, when there are too many incremental data directories, the recovery process feels quite cumbersome. It’s not feasible to restore them day by day, right?
The date of the most recent full backup should only be restored day by day. The backup source data information recorded in the log backup directory cannot be merged.
|
username: TiDBer_BwNZ5U9X
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: