Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: 配置Haproxy透传参数send-proxy后，tidb日志一直报write: connection reset by peer
[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version]
[Reproduction Path] What operations were performed to cause the issue
vim haproxy.cfg
Add send-proxy parameter, restart HAproxy
192.168.1.217: TiDB node
192.168.1.214: HAproxy node
server tidb-192.168.1.217:4000 192.168.1.217:4000 send-proxy check inter 2000 rise 2 fall 3 # Check port 4000, check frequency is every 2000 milliseconds. If 2 checks are successful, the server is considered available; if 3 checks fail, the server is considered unavailable.
[Encountered Issue: Problem Phenomenon and Impact]
tidb.log continuously logs the following:
[2023/06/12 09:17:04.062 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636319] [error="write tcp 192.168.1.217:4000->192.168.1.214:39392: write: connection reset by peer"] ["remote addr"=192.168.1.214:39392]
[2023/06/12 09:17:06.063 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636321] [error="write tcp 192.168.1.217:4000->192.168.1.214:39416: write: connection reset by peer"] ["remote addr"=192.168.1.214:39416]
[2023/06/12 09:17:08.064 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636323] [error="write tcp 192.168.1.217:4000->192.168.1.214:39438: write: connection reset by peer"] ["remote addr"=192.168.1.214:39438]
[2023/06/12 09:17:10.065 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636325] [error="write tcp 192.168.1.217:4000->192.168.1.214:39462: write: connection reset by peer"] ["remote addr"=192.168.1.214:39462]
[2023/06/12 09:17:12.068 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636327] [error="write tcp 192.168.1.217:4000->192.168.1.214:39484: write: connection reset by peer"] ["remote addr"=192.168.1.214:39484]
[2023/06/12 09:17:14.069 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636329] [error="write tcp 192.168.1.217:4000->192.168.1.214:39510: write: connection reset by peer"] ["remote addr"=192.168.1.214:39510]
[2023/06/12 09:17:16.071 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636331] [error="write tcp 192.168.1.217:4000->192.168.1.214:39530: write: connection reset by peer"] ["remote addr"=192.168.1.214:39530]
[2023/06/12 09:17:18.078 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636333] [error="write tcp 192.168.1.217:4000->192.168.1.214:39556: write: connection reset by peer"] ["remote addr"=192.168.1.214:39556]
[2023/06/12 09:17:20.076 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636335] [error="write tcp 192.168.1.217:4000->192.168.1.214:39576: write: connection reset by peer"] ["remote addr"=192.168.1.214:39576]
[2023/06/12 09:17:22.077 +08:00] [WARN] [server.go:644] ["Server.onConn handshake"] [conn=3413847264802636337] [error="write tcp 192.168.1.217:4000->192.168.1.214:39602: write: connection reset by peer"] ["remote addr"=192.168.1.214:39602]
I think the main reason is that the TiDB version is too old.
Is this parameter configured on the TiDB cluster?
This is necessary, otherwise how can it be passed through~
It doesn’t have much impact, you can ignore it for now.
After changing to 10080, there is no error, but using the VIP link will report an error:
ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0
It can be used, but the logs are generated too frequently.
Is there an issue with the VIP configuration? This only changes the default status check port from 4000 to TiDB’s stats port 10080, which shouldn’t have any impact. If the VIP could connect before, it should still be able to connect after changing this port.
Change to 4000, and it can connect normally using VIP. Change to 10080, TiDB does not report an error, but VIP connection reports an error, direct connection is fine.
ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0
proxy-protocol.networks: Configured HAproxy’s IP.
HAProxy version:
haproxy -v
HAProxy version 2.6.2-16a3646 2022/07/22 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.2.html
Running on: Linux 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64
haproxy.cfg content is as follows:
[tidb@centos-g haproxy]$ cat haproxy.cfg
global # Global configuration.
log 127.0.0.1 local2 # Define the global syslog server, up to two can be defined.
chroot /opt/haproxy/var/lib/haproxy # Change the current directory and set superuser privileges for the startup process to enhance security.
pidfile /opt/haproxy/var/run/haproxy.pid # Write the PID of the HAProxy process to the pidfile.
maxconn 4096 # Maximum number of concurrent connections a single HAProxy process can accept, equivalent to the command line parameter "-n".
nbthread 48 # Maximum number of threads. The upper limit of the number of threads is the same as the number of CPUs.
user tidb # Same as the UID parameter.
group tidb # Same as the GID parameter, it is recommended to use a dedicated user group.
daemon # Let HAProxy work in the background as a daemon process, equivalent to the command line parameter "-D". Of course, it can also be disabled with the "-db" parameter on the command line.
stats socket /opt/haproxy/var/lib/haproxy/stats # Location to save statistics.
defaults # Default configuration.
log global # Logs inherit the settings of the global configuration section.
retries 2 # Maximum number of attempts to connect to the upstream server, exceeding this value will consider the backend server unavailable.
timeout connect 2s # Timeout for HAProxy to connect to the backend server. If within the same LAN, it can be set to a shorter time.
timeout client 30000s # Timeout for inactive connections after the client connects to HAProxy and data transfer is complete.
timeout server 30000s # Timeout for inactive connections on the server side.
listen admin_stats # Combination of frontend and backend, the name of this monitoring group can be customized as needed.
bind 0.0.0.0:8080 # Listening port.
mode http # Mode in which the monitoring runs, here it is `http` mode.
option httplog # Enable logging of HTTP requests.
maxconn 10 # Maximum number of concurrent connections.
stats refresh 30s # Automatically refresh the monitoring page every 30 seconds.
stats uri /haproxy # URL of the monitoring page.
stats realm HAProxy # Prompt information on the monitoring page.
stats auth admin:Pin # User and password for the monitoring page, multiple usernames can be set.
stats hide-version # Hide the HAProxy version information on the monitoring page.
stats admin if TRUE # Manually enable or disable backend servers (supported from HAProxy 1.4.9 onwards).
listen tidb-cluster # Configure database load balancing.
bind 192.168.1.238:4000 # Floating IP and listening port.
mode tcp # HAProxy will use the 4th layer transport layer.
balance leastconn # The server with the least number of connections receives the connection first. `leastconn` is recommended for long session services such as LDAP, SQL, TSE, etc., rather than short session protocols like HTTP. This algorithm is dynamic, and the server weight will be adjusted during operation for slow-starting servers.
server tidb-192.168.1.217:4000 192.168.1.217:10080 send-proxy check inter 2000 rise 2 fall 3 # Check port 4000, check frequency is once every 2000 milliseconds. If 2 checks are successful, the server is considered available; if 3 checks fail, the server is considered unavailable.
server tidb-192.168.1.237:4000 192.168.1.237:10080 send-proxy check inter 2000 rise 2 fall 3
proxy-protocol.networks:
You have a configuration issue.
Try my example (although I didn’t use VIP).
The port should still be 4000. Just change the check port to the status port of TiDB (default is 10080).
Thank you very much. By default, there is no check port. Adding a check port 10080 will solve it.
server tidb-192.168.1.217:4000 192.168.1.237:4000 check port 10080 send-proxy inter 2000 rise 2 fall 3
Awesome, I’ll give it a try too.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
|
username: 我是咖啡哥
|
username: 
Operations Guide / TiDB Common FAQ - TiDB Q&A Community (asktug.com)
|
username: 
|
username: 
|
username: 
I have the same problem.
|
username: