TiDB Log Error: Regarding HAProxy

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb日志报错:关于HAProxy

| username: TiDBer_小阿飞

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.0
[Reproduction Path] Deploy HAProxy, then TiDB node logs keep reporting warnings. How can I stop these logs from appearing?
[Encountered Problem: Phenomenon and Impact]
Log file: tidb.log
Log content:

[WARN] [server.go:614] ["Server.onConn handshake"] [conn=3919*********] [error="Invalid PROXY Protocol Header"] ["remote addr="21.72.***.**:42246"]
[WARN] [server.go:614] ["Server.onConn handshake"] [conn=3916*********] [error="Invalid PROXY Protocol Header"] ["remote addr="21.72.***.**:42990"]


Environment: 4 TiDB nodes, 1 HAProxy. HA address configuration: 0.0.0.0:13390
haproxy.cfg configuration is as follows:

global
    log            127.0.0.1 local1
    chroot         /var/lib/haproxy
    pidfile        /var/run/haproxy.pid
    maxconn        4096
    nbthread       48
    user           haproxy
    daemon

defaults
    log                 global
    retries             2
    timeout connect     2s
    timeout client      30000s
    timeout server      30000s

listen admin_stats
    bind                0.0.0.0:8000
    mode                http
    maxconn             10
    stats refresh       30s
    stats uri           /haproxy
    stats realm         HAProxy
    stats auth          admin:admin
    stats hide-version
    stats admin if TRUE

listen tidb-cluster
    bind                0.0.0.0:13390
    mode                tcp
    balance             leastconn
    server tidb-1 21.72.***.*1:4000 send-proxy check port 10080 inter 2000 rise 2 fall 3
    server tidb-2 21.72.***.*2:4000 send-proxy check port 10080 inter 2000 rise 2 fall 3
    server tidb-3 21.72.***.*3:4000 send-proxy check port 10080 inter 2000 rise 2 fall 3  
    server tidb-4 21.72.***.*4:4000 send-proxy check port 10080 inter 2000 rise 2 fall 3
    .......
| username: 大飞哥online | Original post link

Here’s the document you requested: HAProxy 在 TiDB 中的最佳实践 | PingCAP 文档中心

| username: zhanggame1 | Original post link

Here’s a document for you to review; some configurations need to be adjusted:

| username: TiDBer_小阿飞 | Original post link

I’ve seen this, but it doesn’t solve the problem of the backend logs continuously being generated. Moreover, checking the port is an adjustment made later; otherwise, another log will keep being generated.

| username: TiDBer_小阿飞 | Original post link

This document is very clear, but it doesn’t explain how to handle the tidb.log logs on the TiDB side.

| username: Fly-bird | Original post link

It should be an issue with your HAproxy configuration check. Please check the configuration for “Invalid PROXY Protocol Header”.

| username: redgame | Original post link

Has it affected usage?

| username: TiDBer_小阿飞 | Original post link

It doesn’t affect usage, but the logs keep running. Over time, they will take up space, and eventually, you’ll have to clear the logs!

| username: 有猫万事足 | Original post link

In the documentation, it is configured like this:

server tidb-1 10.9.18.229:4000 send-proxy check inter 2000 rise 2 fall 3

In your configuration, you added an extra port 10080. Try removing it and see if it works. Actually, this log indicates that the wrong port was given for the health check, which caused the issue.

| username: TiDBer_小阿飞 | Original post link

Here, the detection port was changed to 10080 instead of 4000 because if not changed, tidb.log would continuously log “write: connection reset by peer.” Initially, it was configured according to the documentation, but after adding the send-proxy parameter, it kept logging “write: connection reset by peer” at a much faster rate than my current warn. Later, changing the port to 10080 stopped the “write: connection reset by peer” logs! However, the error “Invalid PROXY Protocol Header” has always been there before and after the adjustment, and I don’t know where to turn it off.

| username: buptzhoutian | Original post link

Looking at this error, it seems that tidb-server does not recognize the proxy protocol.

You have configured haproxy to use the proxy protocol to access tidb-server, so you need to make tidb-server also accept the proxy protocol.

Reference documentation:

| username: zhanggame1 | Original post link

I feel like this kind of prompt indicates a protocol mismatch. Maybe try using a different version of HAProxy?