Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: haproxy 代理tidb 链接数到1000后，无法登录
[TiDB Usage Environment] Test
[TiDB Version] v6.1.1
[Reproduction Path]
[Encountered Issue: Symptoms and Impact] From the monitoring, it can be seen that after the TiDB connection count reaches 1000, logging in with mysql -h haproxyip -P 3390 -u root -p and entering the password causes it to hang. However, telnet haproxyip 3390 works, and directly logging in with mysql -h tidbip -P 4000 -u root -p is successful. The show processlist; command shows the connection count as 1001.
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
HAProxy configuration is as follows:
global
log 127.0.0.1 local0 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 40960
user pirate
group pirate
daemon
stats socket /var/lib/haproxy/stats
defaults
log global
retries 2
timeout connect 2s
timeout client 30000s
timeout server 30000s
listen admin_stats
bind 0.0.0.0:8080
mode http
stats enable
option httplog
maxconn 10
stats refresh 30s
stats uri /haproxy
stats realm HAProxy
stats auth admin:pingcap123
stats hide-version
stats admin if TRUE
listen tidb-cluster
bind 0.0.0.0:3390
mode tcp
balance leastconn
server tidb-1 192.168.1.1:4000 check inter 2000 rise 2 fall 3
#server tidb-2 192.168.1.180:4000 check inter 2000 rise 2 fall 3
server tidb-3 192.168.2.1:4000 check inter 2000 rise 2 fall 3
Check if the server settings are too low.
It doesn’t seem like it. If it were a server-level file opening limit or something similar, telnet shouldn’t work either. However, I haven’t confirmed the server-level settings optimization. I’ll optimize it first and give it a try.
Sure, please provide the Chinese text you need translated.
After taking a closer look, it seems like I can’t connect to TiDB. Although telnet is accessible, the return is abnormal.
Here is the telnet return when there is an issue:
telnet 192.168.2.1 3390
Trying 192.168.2.1...
Connected to 192.168.2.1.
Escape character is '^]'.
^]
telnet> Connection closed.
Here is the normal telnet return:
telnet 192.168.2.1 3390
Trying 192.168.2.1...
Connected to 192.168.2.1.
Escape character is '^]'.
V
5.7.25-TiDB-v6.1.1˧J;n Fs¦.9bmN44]ymysql_native_password^CConnection closed by foreign host.
It looks a bit like an error reported by TiDB. Check the connection number configuration?
It looks like maxconn is greater than 1000, check if there are any overrides in other places.
It looks like max_connections is set to 10000.
MySQL [(none)]> show variables like ‘%max_connections%’;
±----------------±------+
| Variable_name | Value |
±----------------±------+
| max_connections | 10000 |
±----------------±------+
1 row in set (0.00 sec)
ps shows that haproxy is using /etc/haproxy/haproxy.cfg, and /etc/sysconfig/haproxy is empty,
root 38226 1 0 Jul12 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
pirate 38227 38226 0 Jul12 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
pirate 38228 38227 1 Jul12 ? 00:18:51 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
cat /etc/sysconfig/haproxy
# Add extra options to the haproxy daemon here. This can be useful for
# specifying multiple configuration files with multiple -f options.
# See haproxy(1) for a complete list of options.
OPTIONS=""
It should be necessary to add maxconn 35000 in both the defaults and listen sections. If not added, the default is 2000.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
|
username: suqingbin0315
|
username: 
|
username: 
|
username: 
|
username: 
|
username: