TiDB Node Error [terror.go:324] ["encountered error"]

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 节点报错[terror.go:324] [“encountered error”]

| username: magdb

[TiDB Usage Environment] Production Environment
[TiDB Version] v7.1.3
[Reproduction Path] TiDB dashboard → Log Search → tidb error
[Encountered Issue: Symptoms and Impact] A large number of error logs appeared on the TiDB node, [terror.go:324] [“encountered error”] [error=“close tcp 192.168.1.xx:4000->192.168.1.xxx:6364: use of closed network connection”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:324\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:647”].
There is haproxy and keepalived for high availability in front of the TiDB node, not sure if it is related to this.
[Resource Configuration]

[Attachments: Screenshots/Logs/Monitoring]

| username: Jellybean | Original post link

The general meaning of this error is that the client accessed the tidb-server, the TCP connection was successfully established, and then when the tidb-server needed to return results or check the connection status, it found that the connection was closed by the client. Therefore, the tidb-server logged an error, indicating that an abnormal situation was encountered.

It looks like the frontend load balancing layer is performing a health check heartbeat on the backend tidb-server upstream. Please confirm if the load balancing component has this logic.

Secondly, you can confirm whether the business is affected.

| username: Kongdom | Original post link

:thinking: Could it be caused by too many connections?

| username: dba远航 | Original post link

The original network connection has been closed, but the node is still using the original connection, resulting in an error. There might be an abnormal disconnection situation.

| username: tidb菜鸟一只 | Original post link

This should be an HAProxy configuration issue, could you share your HAProxy configuration?

| username: magdb | Original post link

Sure, here is our haproxy configuration. 20/21 are the backend TiDB nodes, and haproxy is also deployed on these two nodes.

| username: tidb菜鸟一只 | Original post link

I remember that if haproxy is used to check tidb, it cannot check port 4000, it needs to check port 10080. Try configuring it like this… check port 10080.

| username: magdb | Original post link

Okay, I’ll give it a try, thanks :grinning:

| username: kkpeter | Original post link

Have you encountered any pitfalls using HAProxy online?

| username: magdb | Original post link

According to the official configuration, there were basically no issues. I’m not sure if this log counts as a pitfall. :joy:

| username: gcworkerishungry | Original post link

You can try connecting directly to the TiDB node.

| username: magdb | Original post link

Direct connection requires business support. Currently, there is no impact on the business, so we don’t want to change to direct connection either. :rofl:

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.