Issue with IP Transparency in TiDB+HAProxy Configuration

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB+HAProxy配置IP透传问题

| username: 源de爸

[Test Environment for TiDB] Testing
[TiDB Version] 7.5
[Reproduction Path] According to the documentation: HAProxy 在 TiDB 中的最佳实践 | PingCAP 文档中心, add the proxy-protocol.networks parameter, and configure the IP inside as the HAproxy node IP.
[Encountered Issues: Phenomenon and Impact]
Issue 1:
When logging into the TiDB machine and using the mysql command to connect to port 4000, it prompts ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading authorization packet’, system error: 0. However, remote connections from other hosts to this TiDB’s port 4000 can access normally. Connections through HAProxy configured with the send-proxy flag also work fine.

Issue 2:
In the dashboard, multiple module pages (slow queries, SQL statement analysis, Top SQL, host tab in cluster information, etc.) all prompt: commands out of sync. You can’t run this command now.

| username: tidb菜鸟一只 | Original post link

You didn’t configure the HAproxy node IP on your TiDB machine, did you? After configuring this address, it can only be accessed through the HAproxy protocol and not through the original TiDB protocol.

| username: 源de爸 | Original post link

HAProxy and TiDB share the same machine.

| username: 源de爸 | Original post link

The main issue is that server resources are quite tight, so we have to manage it this way.

| username: tidb菜鸟一只 | Original post link

Then haproxy doesn’t make much sense. Normally, it should be deployed on other machines and load balanced to 3 tidb-servers. If one tidb-server goes down, it won’t affect usage. In your case, if it goes down, both haproxy and tidb-server will be down…

| username: 源de爸 | Original post link

That’s not the case. I am planning to deploy HAProxy and TiDB on multiple machines, and then use Keepalived to implement VIP drift among these machines to achieve high availability.

| username: Jolyne | Original post link

Is this configured?

| username: 源de爸 | Original post link

Configured, I wrote it in the [Reproduction Path] section of the issue.

| username: tidb菜鸟一只 | Original post link

After this configuration, the corresponding IP can only be accessed through the proxy protocol and not through the TiDB protocol. Therefore, your dashboard will not work. You might want to deploy HAProxy on a different machine or avoid IP passthrough.

| username: Billmay表妹 | Original post link

Question 1:
Take a look at this:

| username: Billmay表妹 | Original post link

Here’s a practical article you can check out: 专栏 - Haproxy 探活 TiDB in Action | TiDB 社区

| username: 源de爸 | Original post link

Looking at the error, it seems to be related to authentication, and the issue is consistently reproducible, so it doesn’t seem like a timeout.

| username: 源de爸 | Original post link

This liveness probe strategy is more detailed than the official documentation. I’ll configure it and give it a try.

| username: 源de爸 | Original post link

Okay, I’ll try deploying it independently first to see the results.

| username: 源de爸 | Original post link

Excuse me, do you have any actual production configuration experience? I would like to ask for some advice.

| username: Jolyne | Original post link

Previously, I also encountered issues with HAProxy. I referred to this, you can take a look. Also, Brother Mao had previously reminded that if there are problems with the apt installation on Ubuntu 20 or later, manual compilation is required.

| username: DBAER | Original post link

Abnormal deployment, you can use the app host.

| username: zhanggame1 | Original post link

I installed Ubuntu 22.04 without any issues. Everything is new and has been running for more than half a year.
HAProxy was installed via apt with version number 2.4.22.

HAProxy also proxies the dashboard and Grafana.

| username: Jolyne | Original post link

I previously used Ubuntu 20, and indeed, after about four or five months, it suddenly couldn’t connect. Later, I manually compiled it, and it worked fine.

| username: zhanggame1 | Original post link

There might still be a bug. Just use the latest version.