Haproxy Load Balancing Error Code 8052

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Haproxy负载出现8052错误编码

| username: TiDBer_cZ23QmYB

【TiDB Usage Environment】Testing
【TiDB Version】5.7.25-TiDB-v7.1.2
【Reproduction Path】None
【Encountered Issues: Phenomenon and Impact】

【Resource Configuration】

  1. All virtual servers are 4C16G
  2. TiDB 环境与系统配置检查 | PingCAP 文档中心 All settings are according to the documentation
  3. Cluster Configuration
    1700645556575
    【Attachments: Screenshots/Logs/Monitoring】
  4. Logs after enabling haproxy check inner configuration
    [2023/11/22 15:23:09.866 +08:00] [WARN] [server.go:644] [“Server.onConn handshake”] [conn=6826990842163536645] [error=“write tcp 192.168.100.181:4000->192.168.100.184:51782: write: connection reset by peer”] [“remote addr”=192.168.100.184:51782]
  5. Navicat connection error prompt and tidb.log log prompt both show error code 8052
  6. Internal exception in the Java application

    Also located in the tidb log is error code 8052
| username: Jolyne | Original post link

Can you share your haproxy configuration?

| username: tidb菜鸟一只 | Original post link

Can the MySQL client log in through the 184 address?

| username: TiDBer_cZ23QmYB | Original post link

global                                     # Global configuration.
   log         127.0.0.1 local2            # Define the global syslog server, up to two can be defined.
   chroot      /var/lib/haproxy            # Change the current directory and set superuser privileges for the startup process to enhance security.
   pidfile     /var/run/haproxy.pid        # Write the HAProxy process PID to the pidfile.
   maxconn     4096                        # Maximum concurrent connections a single HAProxy process can accept, equivalent to the command line parameter "-n".
   nbthread    48                          # Maximum number of threads. The upper limit of threads is the same as the number of CPUs.
   user        root#haproxy                     # Same as the UID parameter.
   group       root#haproxy                     # Same as the GID parameter, it is recommended to use a dedicated user group.
   daemon                                  # Let HAProxy work in the background as a daemon, equivalent to the command line parameter "-D". Of course, it can also be disabled with the "-db" parameter on the command line.
   stats socket /var/lib/haproxy/stats     # Location to save statistics.

defaults                                   # Default configuration.
   log global                              # Logs inherit the settings from the global configuration section.
   retries 2                               # Maximum number of attempts to connect to the upstream server, beyond which the backend server is considered unavailable.
   timeout connect  20s                     # Timeout for HAProxy to connect to the backend server. If within the same LAN, a shorter time can be set.
   timeout client 30000s                   # Timeout for inactive connections between the client and HAProxy after data transfer is complete.
   timeout server 30000s                   # Timeout for inactive connections on the server side.

listen admin_stats                         # Combination of frontend and backend, the name of this monitoring group can be customized as needed.
   bind 0.0.0.0:8080                       # Listening port.
   mode http                               # Mode in which the monitoring runs, here it is `http` mode.
   option httplog                          # Enable logging of HTTP requests.
   maxconn 10                              # Maximum concurrent connections.
   stats refresh 30s                       # Automatically refresh the monitoring page every 30 seconds.
   stats uri /haproxy                      # URL of the monitoring page.
   stats realm HAProxy                     # Prompt information on the monitoring page.
   stats auth admin:pingcap123             # User and password for the monitoring page, multiple usernames can be set.
   stats hide-version                      # Hide the HAProxy version information on the monitoring page.
   stats  admin if TRUE                    # Manually enable or disable backend servers (supported from HAProxy 1.4.9 onwards).

listen tidb-cluster                        # Configure database load balancing.
   bind 0.0.0.0:4001                       # Floating IP and listening port.
   mode tcp                                # HAProxy should use the 4th layer transport layer.
   balance leastconn                       # The server with the least connections receives the connection first. `leastconn` is recommended for long-session services such as LDAP, SQL, TSE, etc., rather than short-session protocols like HTTP. This algorithm is dynamic, and the server weight will be adjusted during operation for slow-starting servers.
   server tidb-1 192.168.100.181:4000 send-proxy       # Check port 4000, with a check frequency of once every 2000 milliseconds. If 2 checks are successful, the server is considered available; if 3 checks fail, the server is considered unavailable.
   server tidb-2 192.168.100.182:4000 send-proxy
| username: TiDBer_cZ23QmYB | Original post link

Navicat, DBeaver, and MySQL Workbench all prompt “invalid sequence 88!=1”.

| username: Jolyne | Original post link

How about writing the last column like this?

| username: TiDBer_cZ23QmYB | Original post link

No effect.

| username: Billmay表妹 | Original post link

First, thumbs up to you. The content of your post is very detailed, and many friends should learn from you~

| username: 有猫万事足 | Original post link

Could it be that you are using Ubuntu 20 and installed HAProxy via apt? The apt source for this version might have issues. You may need to compile from source and reinstall.

| username: Jolyne | Original post link

I encountered this issue last time, fortunately, Brother Cat helped me solve it.

| username: TiDBer_cZ23QmYB | Original post link

HAProxy 2.6.2
CentOS: Linux version 3.10.0-1160.102.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Oct 17 15:42:21 UTC 2023
Installed according to the documentation HAProxy 在 TiDB 中的最佳实践 | PingCAP 文档中心

I will try reinstalling it once more.

| username: 有猫万事足 | Original post link

If it’s CentOS, there shouldn’t be much of a problem.

| username: ShawnYan | Original post link

Has the issue been resolved? The documentation is for compiling HAProxy from source; is that the case in your environment as well? Which version of JDBC is configured in your Java setup?

| username: dba远航 | Original post link

Upgrade the version of Navicat. It’s mostly caused by the version being too low and not supported.

| username: heiwandou | Original post link

It looks like the client failed to execute some initial SQL when connecting to the database. You can try using a native version or try using MySQL Workbench.

| username: Kongdom | Original post link

Will there be an error if you connect directly to TiDB without using a load balancer? Excluding errors from the database itself.

| username: 随缘天空 | Original post link

Try referring to this article: 专栏 - HAProxy安装及搭建tidb数据库负载均衡服务实战 | TiDB 社区

| username: TiDBer_小阿飞 | Original post link

There is no check port check inter
Additionally, set fallbackable to true to avoid errors when using a single node later.

| username: TiDBer_cZ23QmYB | Original post link

(server: Fix compression protocol for larger packets | tidb-test=pr/2227 by dveeden · Pull Request #47495 · pingcap/tidb · GitHub)

| username: TiDBer_cZ23QmYB | Original post link

References:

  1. https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
  2. server: Fix compression protocol for larger packets | tidb-test=pr/2227 by dveeden · Pull Request #47495 · pingcap/tidb · GitHub
    When using proxy configuration, there is an issue with the TiDB server-side code. Remove the send-proxy configuration.