When the number of connections exceeds dozens, TiDB will freeze and become unresponsive

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 当连接超过几十，tidb就会卡死无法连接

| username: TiDBer_QddfzfEg

The TiDB version is 5.2;
During the sysbench test, when the thread is set to 16, it is not possible to connect to the TiDB service through Navicat.

image
image666×1000 95.4 KB

image

2
| username: Meditator | Original post link

Try connecting manually using MySQL Shell. If it works, then the issue lies with the UI tool.

3
| username: alfred | Original post link

Are there any error logs in TiDB?

4
| username: TiDBer_QddfzfEg | Original post link

It doesn’t seem like it. I also can’t connect due to a timeout when I try to connect through the program.

5
| username: TiDBer_QddfzfEg | Original post link

There is no error in tidb_stderr.log, but there is an error in tidb.log:
[2022/08/25 10:36:35.048 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[server:1045]Access denied for user ‘root’@‘16.16.16.65’ (using password: NO)”] [stack=“github.com/pingcap/tidb/parser/terror.Log
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307
github.com/pingcap/tidb/server.(*Server).onConn
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
This is because 16.16.16.65 was previously a drainer. I configured binlog, but felt the concurrency was too low, so I paused pump and drainer, which caused this error.

6
| username: TiDBer_QddfzfEg | Original post link

[2022/08/25 00:22:37.237 +08:00] [ERROR] [analyze.go:177] [“save table stats to storage failed”] [error=“previous statement: update mysql.stats_histograms set last_analyze_pos = _binary’\u0003\ufffd\0\0\0\0}\u0018\ufffd’ where table_id = 596821 and is_index = 1 and hist_id = 1: tikv aborts txn: Error(Txn(Error(Mvcc(Error(PessimisticLockNotFound { start_ts: TimeStamp(435514798432321537), key: [116, 128, 0, 0, 0, 0, 0, 0, 25, 95, 105, 128, 0, 0, 0, 0, 0, 0, 1, 3, 128, 0, 0, 0, 0, 9, 27, 85, 3, 128, 0, 0, 0, 0, 0, 0, 0, 3, 128, 0, 0, 0, 0, 0, 0, 2, 3, 128, 0, 0, 0, 0, 0, 0, 247] })))))”]
[2022/08/25 00:22:37.237 +08:00] [ERROR] [update.go:1191] [“[stats] auto analyze failed”] [sql=“analyze table sbtest.sbtest2 index k_2”] [cost_time=17m29.807513151s] [error=“previous statement: update mysql.stats_histograms set last_analyze_pos = _binary’\u0003\ufffd\0\0\0\0}\u0018\ufffd’ where table_id = 596821 and is_index = 1 and hist_id = 1: tikv aborts txn: Error(Txn(Error(Mvcc(Error(PessimisticLockNotFound { start_ts: TimeStamp(435514798432321537), key: [116, 128, 0, 0, 0, 0, 0, 0, 25, 95, 105, 128, 0, 0, 0, 0, 0, 0, 1, 3, 128, 0, 0, 0, 0, 9, 27, 85, 3, 128, 0, 0, 0, 0, 0, 0, 0, 3, 128, 0, 0, 0, 0, 0, 0, 2, 3, 128, 0, 0, 0, 0, 0, 0, 247] })))))”]
[2022/08/25 00:39:26.633 +08:00] [ERROR] [binloginfo.go:251] [“write binlog failed”] [binlog_type=Prewrite] [binlog_start_ts=435515337361063937] [binlog_commit_ts=0] [error=“write binlog failed, the last error no available pump to write binlog”]
[2022/08/25 00:39:26.633 +08:00] [ERROR] [binloginfo.go:257] [“write binlog fail but error ignored”]
[2022/08/25 00:52:08.627 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37028: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*packetIO).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\github.com/pingcap/tidb/server.(*clientConn).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\github.com/pingcap/tidb/server.(*clientConn).writeError\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:08.627 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:11.629 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37054: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*packetIO).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\github.com/pingcap/tidb/server.(*clientConn).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\github.com/pingcap/tidb/server.(*clientConn).writeError\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:11.629 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:16.213 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37012: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*packetIO).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\github.com/pingcap/tidb/server.(*clientConn).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\github.com/pingcap/tidb/server.(*clientConn).writeError\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:16.213 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:25.208 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37068: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*packetIO).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\github.com/pingcap/tidb/server.(*clientConn).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\github.com/pingcap/tidb/server.(*clientConn).writeError\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:25.208 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:28.333 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37042: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*packetIO).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\github.com/pingcap/tidb/server.(*clientConn).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\github.com/pingcap/tidb/server.(*clientConn).writeError\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:28.334 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:31.160 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37006: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*packetIO).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\github.com/pingcap/tidb/server.(*clientConn).writePacket\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\github.com/pingcap/tidb/server.(*clientConn).writeError\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:31.160 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*clientConn).Run\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 01:18:10.403 +08:00] [ERROR] [domain.go:1041] [“LoadSysVarCacheLoop loop watch channel closed”]
[2022/08/25 01:18:10.413 +08:00] [ERROR] [domain.go:978] [“load privilege loop watch channel closed”]
[2022/08/25 01:18:11.527 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=EOF] [stack=“github.com/pingcap/tidb/parser/terror.Log\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\github.com/pingcap/tidb/server.(*Server).onConn\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/08/25 01:18:12.022 +08:00] [ERROR] [client.go:786] [“[pd] getTS error”] [dc-location=global] [error=“[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster”]
[2022/08/25 01:18:12.027 +08:00

7
| username: TiDBer_QddfzfEg | Original post link

The image is not visible. Please provide the text you need translated.