Unable to log in after HAProxy proxy TiDB connections reach 1000

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: haproxy 代理tidb 链接数到1000后,无法登录

| username: suqingbin0315

[TiDB Usage Environment] Test
[TiDB Version] v6.1.1
[Reproduction Path]
[Encountered Issue: Symptoms and Impact] From the monitoring, it can be seen that after the TiDB connection count reaches 1000, logging in with mysql -h haproxyip -P 3390 -u root -p and entering the password causes it to hang. However, telnet haproxyip 3390 works, and directly logging in with mysql -h tidbip -P 4000 -u root -p is successful. The show processlist; command shows the connection count as 1001.
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

[Attachments: Screenshots/Logs/Monitoring]
HAProxy configuration is as follows:

   log local0 debug
   chroot      /var/lib/haproxy
   pidfile     /var/run/haproxy.pid
   maxconn     40960
   user        pirate
   group       pirate
   stats socket /var/lib/haproxy/stats

   log global
   retries 2
   timeout connect  2s
   timeout client 30000s
   timeout server 30000s

listen admin_stats
   mode http
   stats enable
   option httplog
   maxconn 10
   stats refresh 30s
   stats uri /haproxy
   stats realm HAProxy
   stats auth admin:pingcap123
   stats hide-version
   stats  admin if TRUE

listen tidb-cluster
   mode tcp
   balance leastconn
   server tidb-1 check inter 2000 rise 2 fall 3
   #server tidb-2 check inter 2000 rise 2 fall 3
   server tidb-3 check inter 2000 rise 2 fall 3
| username: 啦啦啦啦啦 | Original post link

Check if the server settings are too low.

| username: suqingbin0315 | Original post link

It doesn’t seem like it. If it were a server-level file opening limit or something similar, telnet shouldn’t work either. However, I haven’t confirmed the server-level settings optimization. I’ll optimize it first and give it a try.

| username: yiduoyunQ | Original post link

Sure, please provide the Chinese text you need translated.

| username: suqingbin0315 | Original post link

After taking a closer look, it seems like I can’t connect to TiDB. Although telnet is accessible, the return is abnormal.

Here is the telnet return when there is an issue:

telnet 3390 
Connected to
Escape character is '^]'.
telnet> Connection closed.

Here is the normal telnet return:

telnet 3390 
Connected to
Escape character is '^]'.
5.7.25-TiDB-v6.1.1˧J;n	Fs¦.9bmN44]ymysql_native_password^CConnection closed by foreign host.
| username: ealam_小羽 | Original post link

It looks a bit like an error reported by TiDB. Check the connection number configuration?

| username: redgame | Original post link

It looks like maxconn is greater than 1000, check if there are any overrides in other places.

| username: suqingbin0315 | Original post link

It looks like max_connections is set to 10000.

MySQL [(none)]> show variables like ‘%max_connections%’;
| Variable_name | Value |
| max_connections | 10000 |
1 row in set (0.00 sec)

| username: suqingbin0315 | Original post link

ps shows that haproxy is using /etc/haproxy/haproxy.cfg, and /etc/sysconfig/haproxy is empty,

root     38226     1  0 Jul12 ?        00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
pirate   38227 38226  0 Jul12 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
pirate   38228 38227  1 Jul12 ?        00:18:51 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

cat /etc/sysconfig/haproxy

# Add extra options to the haproxy daemon here. This can be useful for
# specifying multiple configuration files with multiple -f options.
# See haproxy(1) for a complete list of options.
| username: suqingbin0315 | Original post link

It should be necessary to add maxconn 35000 in both the defaults and listen sections. If not added, the default is 2000.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.