The thread hard limit is 10000

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: the thread hard limit 10000

| username: TiDBer_Lee

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version] v7.1.1

When I was testing the TiFlash compute-storage separation solution, I ran a relatively complex SQL and encountered the following error:

  • [err="other error for mpp stream: From MPP<query:<query_ts:1692067368491702029, local_query_id:218, server_id:3873893, start_ts:443565308233908225>,task_id:73>: Code: 0, e.displayText() = DB::Exception: threads are unavailable for the query <query_ts:1692067368491702029, local_query_id:218, server_id:3873893, start_ts:443565308233908225> (is min_query_id <query_ts:1692067368491702029, local_query_id:218, server_id:3873893, start_ts:443565308233908225>) when directly schedule it, need 421, but used 9768 of the thread hard limit 10000, 1 active and 0 waiting queries., e.what() = DB::Exception,\ngithub.com/pingcap/tidb/store/copr.(*mppIterator).handleMPPStreamResponse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/mpp.go:489\ngithub.com/pingcap/tidb/store/copr.(*mppIterator).establishMPPConns\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/mpp.go:447\ngithub.com/pingcap/tidb/store/copr.(*mppIterator).handleDispatchReq\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/mpp.go:353\ngithub.com/pingcap/tidb/store/copr.(*mppIterator).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/mpp.go:194\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"]

This error indicates that the required threads exceeded the limit of 10,000. Could you please advise where this limit is set and what the general principle behind it is?

| username: ShawnYan | Original post link

System settings

| username: tidb菜鸟一只 | Original post link

Directly use the command ulimit -a with the tidb user to check the current parameters.

| username: redgame | Original post link

Is it this one, max-server-connections?

| username: TiDBer_Lee | Original post link

核心文件大小 (块, -c) 0
数据段大小 (千字节, -d) 无限
调度优先级 (-e) 0
文件大小 (块, -f) 无限
挂起信号 (-i) 245488
最大锁定内存 (千字节, -l) 64
最大内存大小 (千字节, -m) 无限
打开文件数 (-n) 1024
管道大小 (512字节, -p) 8
POSIX消息队列 (字节, -q) 819200
实时优先级 (-r) 0
堆栈大小 (千字节, -s) 8192
CPU时间 (秒, -t) 无限
最大用户进程数 (-u) 4096
虚拟内存 (千字节, -v) 无限
文件锁 (-x) 无限

| username: TiDBer_Lee | Original post link

Which specific setting are you referring to?

| username: tidb菜鸟一只 | Original post link

SHOW GLOBAL VARIABLES LIKE ‘%max_connections%’;
Check the maximum number of connections.

| username: zhanggame1 | Original post link

Take a look at this, issues arise with threads above 10,000.
Linux thread limit - Zhihu (zhihu.com)