BR Backup Error: runtime: failed to create new OS thread (have 95 already; errno=11)

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br备份报错runtime: failed to create new OS thread (have 95 already; errno=11)

| username: realcp1018

【TiDB Version】

Release Version: v4.0.9
Git Commit Hash: 804aa908719748888010edaa944757b82f99f4df
Git Branch: heads/refs/tags/v4.0.9
Go Version: go1.13
UTC Build Time: 2020-12-19 04:53:25
Race Enabled: false

【Encountered Problem】
Error when executing br:

runtime: failed to create new OS thread (have 28 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc
runtime: failed to create new OS thread (have 29 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

runtime stack:
runtime.throw(0x234c802, 0x9)
        runtime/panic.go:774 +0x72
runtime.newosproc(0xc0004d4380)
        runtime/os_linux.go:153 +0x1ba
runtime.newm1(0xc0004d4380)
        runtime/proc.go:1853 +0xdc
runtime.newm(0x2417410, 0xc000098500)
        runtime/proc.go:1832 +0x8f
runtime.startm(0x0, 0xc00045e601)
        runtime/proc.go:1969 +0x12a
runtime.wakep(...)
        runtime/proc.go:2050
runtime.resetspinning()
        runtime/proc.go:2430 +0x93
runtime.schedule()
        runtime/proc.go:2531 +0x2a0
runtime.mstart1()
        runtime/proc.go:1208 +0x8e
runtime.mstart()
        runtime/proc.go:1167 +0x6e

goroutine 1 [runnable, locked to thread]:
sync.(*Pool).Put(0x402d740, 0x2292980, 0xc00016c0d0)
        sync/pool.go:90 +0xc4
fmt.(*pp).free(0xc00016c0d0)
        fmt/print.go:161 +0xb1
fmt.Sprintf(0x23432d6, 0x5, 0xc00015fd60, 0x2, 0x2, 0xc00015fd50, 0x40caf8)
        fmt/print.go:221 +0xb4
github.com/pingcap/parser/terror.ErrClass.initError(0xd, 0x475, 0x2, 0xdd275e85d3476a56)
        github.com/pingcap/parser@v0.0.0-20201214100800-9fb44a0518a2/terror/terror.go:138 +0x16d
github.com/pingcap/parser/terror.ErrClass.NewStdErr(0xd, 0x475, 0xc000254db0, 0xc000203560)
        github.com/pingcap/parser@v0.0.0-20201214100800-9fb44a0518a2/terror/terror.go:159 +0x3c
github.com/pingcap/tidb/util/dbterror.ErrClass.NewStd(0xd, 0x475, 0xc000748000)
        github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/util/dbterror/terror.go:55 +0x72
github.com/pingcap/tidb/privilege/privileges.init()
        github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/privilege/privileges/errors.go:24 +0x102

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
        runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        os/signal/signal_unix.go:29 +0x41

runtime stack:
runtime.throw(0x234c802, 0x9)
        runtime/panic.go:774 +0x72
runtime.newosproc(0xc00050c700)
        runtime/os_linux.go:153 +0x1ba
runtime.newm1(0xc00050c700)
        runtime/proc.go:1853 +0xdc
runtime.newm(0x0, 0xc00008ef00)
        runtime/proc.go:1832 +0x8f
runtime.startm(0xc00008ef00, 0x40a0800)
        runtime/proc.go:1969 +0x12a
runtime.handoffp(0xc00008ef00)
        runtime/proc.go:1996 +0x52
runtime.stoplockedm()
        runtime/proc.go:2064 +0xf2
runtime.schedule()
        runtime/proc.go:2469 +0x485
runtime.goschedImpl(0xc000000180)
        runtime/proc.go:2625 +0xd7
runtime.gopreempt_m(0xc000000180)
        runtime/proc.go:2653 +0x34
runtime.newstack()
        runtime/stack.go:1038 +0x299
runtime.morestack()
        runtime/asm_amd64.s:449 +0x8f

During actual backup, it occasionally succeeds. The error sometimes is very long (prints more goroutine information). Manual execution error is as above. ulimit -n is set to 102400.

| username: yilong | Original post link

What is the value of ulimit -u?

| username: realcp1018 | Original post link

tidb is 10240, I used br to execute tidb.
/etc/security/limits.d/20-nproc.conf

*          soft    nproc     10240
root       soft    nproc     unlimited

I will change it to 102400 and observe.

| username: yilong | Original post link

How is it, are there still any errors?

| username: realcp1018 | Original post link

At the moment, it seems there isn’t any. This should be the issue. I will observe for a while.
Thanks for the support!

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.