HAProxy Load Balancing TiDB Reports Error 8052

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: haproxy负载tidb 报8052错误

| username: Hacker_5YEF49J7

[TiDB Usage Environment] Testing
[TiDB Version] 5.4.0

[Encountered Issue: Phenomenon and Impact]
Connecting to the HAProxy proxy reports error 8052, but connecting directly to the backend TiDB 4000 does not report an error. It used to work without errors, but now it reports an error.

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

image
image695×823 19.2 KB

haproxy.cfg

listen tidb_cluster
    bind    0.0.0.0:3161
    mode    tcp
    option  tcplog
    balance roundrobin
    timeout client  3h
    timeout server  3h
    server  tidb1ser  10.34.89.30:4000 send-proxy  check  inter  5000  rise  2  fall  3
    server  tidb2ser  10.34.89.31:4000 send-proxy  check  inter  5000  rise  2  fall  3
    server  tidb3ser  10.34.89.32:4000 send-proxy  check  inter  5000  rise  2  fall  3

image
image1017×854 47.2 KB

2
| username: WalterWj | Original post link

This error :thinking:, sequence? Used sequence? It seems like there is an issue with some sequence?

3
| username: Kongdom | Original post link

Search in the configuration file to see if there is 88.

4
| username: Hacker_5YEF49J7 | Original post link

Which configuration file should I search for? We upgraded from 5.2.1 to 5.4.0.

find /tidb/app -type f -name “*.toml” | xargs grep “88”
Didn’t find 88.

5
| username: tomsence | Original post link

How about sharing the TiDB log?

6
| username: Hacker_5YEF49J7 | Original post link

Need that log.

7
| username: tomsence | Original post link

tidb.log

8
| username: Hacker_5YEF49J7 | Original post link

[2022/11/24 09:40:03.177 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55248: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:08.178 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55284: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:13.177 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55326: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:18.178 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55372: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:23.179 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55412: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:28.181 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55454: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:33.181 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55490: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:38.182 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55524: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:43.183 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55564: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:40:48.184 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:55600: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]

9
| username: tomsence | Original post link

This log seems to be all heartbeats, no impact. Try to simulate it and capture the logs again.

10
| username: Hacker_5YEF49J7 | Original post link

[2022/11/24 09:52:03.405 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32786: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:08.407 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32826: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:08.735 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[server:8052]invalid sequence 88 != 1”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:13.409 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32872: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:13.624 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[server:8052]invalid sequence 88 != 1”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/11/24 09:52:18.410 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 10.3.87.30:4000->10.3.87.36:32928: write: connection reset by peer”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]

11
| username: tomsence | Original post link

It still seems to be an issue with the HAProxy configuration. Refer to this and check again: HAProxy Best Practices in TiDB | PingCAP Docs
Also, check the firewall, permissions, etc.

12
| username: Hacker_5YEF49J7 | Original post link

proxy-protocol.networks: This parameter disappeared after the upgrade. It will be fine once configured.

13
| username: Minorli-PingCAP | Original post link

This is a pass-through parameter.