K8S BR Error BR:Stream:ErrStreamLogTaskExist

translator_bot · June 22, 2024, 5:11am

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: K8S BR报错 BR:Stream:ErrStreamLogTaskExist

| username: EricSong

[TiDB Usage Environment]
Testing
[TiDB Version]
v6.5.0
[Reproduction Path]
Deploy Backup Schedule resource according to the official documentation. The Backup Schedule includes a snapshot backup and a log backup.
[Encountered Issue: Problem Phenomenon and Impact]
The snapshot backup runs normally, but the log backup fails. According to the pod logs, there is an existing running log backup job. However, using kubectl get bk -n tidb-dev only prints the current tasks (snapshot, log).
[Attachment: Screenshot/Logs/Monitoring]
Log-backup Pod Error Logs

I0627 09:39:57.519515       9 backup.go:262] [2023/06/27 09:39:57.519 +00:00] [ERROR] [main.go:59] ["br failed"] [error="It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists"] [errorVerbose="[BR:Stream:ErrStreamLogTaskExist]stream task already exists\nIt supports single stream log task currently\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamStart\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:550\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:506\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamStartCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
I0627 09:39:57.536465       9 backup.go:262] 
I0627 09:39:57.536481       9 backup.go:269] Error: It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists
E0627 09:39:57.536558       9 manager.go:479] Start log backup of cluster tidb-dev/log-kube-tidb-backup failed, err: cluster tidb-dev/log-kube-tidb-backup, wait pipe message failed, errMsg [2023/06/27 09:39:57.518 +00:00] [ERROR] [stream.go:507] ["failed to stream"] [command="log start"] [error="It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists"] [errorVerbose="[BR:Stream:ErrStreamLogTaskExist]stream task already exists\nIt supports single stream log task currently\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamStart\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:550\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:506\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamStartCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="github.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:507\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamStartCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"]
[2023/06/27 09:39:57.519 +00:00] [ERROR] [main.go:59] ["br failed"] [error="It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists"] [errorVerbose="[BR:Stream:ErrStreamLogTaskExist]stream task already exists\nIt supports single stream log task currently\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamStart\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:550\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:506\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamStartCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Error: It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists
, err: exit status 1
I0627 09:39:57.548654       9 backup_status_updater.go:123] Backup: [tidb-dev/log-tidb-backup] updated successfully
error: cluster tidb-dev/log-tidb-backup, wait pipe message failed, errMsg [2023/06/27 09:39:57.518 +00:00] [ERROR] [stream.go:507] ["failed to stream"] [command="log start"] [error="It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists"] [errorVerbose="[BR:Stream:ErrStreamLogTaskExist]stream task already exists\nIt supports single stream log task currently\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamStart\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:550\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:506\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamStartCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/06/27 09:39:57.519 +00:00] [ERROR] [main.go:59] ["br failed"] [error="It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists"] [errorVerbose="[BR:Stream:ErrStreamLogTaskExist]stream task already exists\nIt supports single stream log task currently\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamStart\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:550\ngithub.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:506\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamStartCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"]
Error: It supports single stream log task currently: [BR:Stream:ErrStreamLogTaskExist]stream task already exists
, err: exit status 1

Existing Backup Template Investigation

kubectl get bk -n tidb-dev
NAME                              TYPE   MODE       STATUS     BACKUPPATH                                                                           BACKUPSIZE   COMMITTS                            LOGTRUNCATEUNTIL   AGE
log-tidb-backup                          log        Failed     s3://dev-us-east-1/tidb-log-backup/log-2023-06-27t09-39-34                                                                                               20m
tidb-backup-2023-06-27t09-40-00   full   snapshot   Complete   s3://dev-us-east-1/tidb-full-backup/tidb-perf-pd.tidb-dev-2379-2023-06-27t09-40-00   18 GB        442462065254989825                      19m
tidb-backup-2023-06-27t09-50-00   full   snapshot   Complete   s3://dev-us-east-1/tidb-full-backup/tidb-perf-pd.tidb-dev-2379-2023-06-27t09-50-00   18 GB        442462222279245825                      9m51s

translator_bot · June 22, 2024, 5:11am

| username: Billmay表妹 | Original post link

Based on your description, it might be due to your log backup Job not being properly deleted, causing the new log backup Job to fail to run correctly. You can resolve this issue by following these steps:

Use the kubectl get job -n <namespace> command to view all Jobs in the current namespace and find the Job related to log backup.
If there is a running log backup Job, you can manually delete it using the kubectl delete job <job-name> -n <namespace> command.
If the log backup still fails to run after deleting the Job, you can try deleting the CronJob and BackupSchedule resources related to the log backup and then recreate these resources.

Before performing the above steps, it is recommended to back up the relevant data to avoid data loss due to misoperation. Additionally, if you still cannot resolve the issue, it is advisable to provide more log information and related configuration details so that we can better assist you in solving the problem.

translator_bot · June 22, 2024, 5:11am

| username: yiduoyunQ | Original post link

Refer to 使用 BR 备份 TiDB 集群数据到兼容 S3 的存储 | PingCAP 文档中心 to stop the current log backup first.

translator_bot · June 22, 2024, 5:11am

| username: Anna | Original post link

Delete job logs

translator_bot · June 22, 2024, 5:11am

| username: EricSong | Original post link

My log backup seems to have failed before it even started, so it still fails after stopping. The main issue is that I can’t find the existing log backup job mentioned in the above logs. Using the following command, I can only find the already failed log backup tasks, and there are no other log backup tasks.

kubectl get backup -n tidb-dev
NAME                                   TYPE   MODE       STATUS     BACKUPPATH                                                                                                   BACKUPSIZE   COMMITTS             LOGTRUNCATEUNTIL   AGE
log-kube-tidb-backup                          log        Failed     s3://dev-us-east-1/tidb-log-backup/log-2023-06-27t09-39-34                                                                                               17h

I checked the logs in the controller-manager as follows:

E0628 02:54:25.593199       1 backup_manager.go:105] backup tidb-dev/log-kube-tidb-backup wait pre task done error log backup tidb-dev/log-kube-tidb-backup command log-stop should wait log backup start complete.

It indeed looks like it failed before it even started, so it can’t be stopped either.

translator_bot · June 22, 2024, 5:11am

| username: yiduoyunQ | Original post link

When starting the log backup, it included name tidb-operator/cmd/backup-manager/app/backup/backup.go at master · pingcap/tidb-operator · GitHub. If you still remember the name, you can directly use the same name to start a stop task with logStop: true. If you don’t remember the name

How about trying to run a bare br to check it out?

Or search the TiKV log to see if there is any relevant information

translator_bot · June 22, 2024, 5:11am

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.