TiDB 7.1.0 Connection Closure Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 7.1.0连接关闭错误

| username: polars

[TiDB Usage Environment] Production Environment / Test / Poc
Test
[TiDB Version]
7.1.0
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
SQL error, application side error connection closed
[ERROR] [conn.go:1072] [“connection running loop panic”] [conn=1669219179662106373] [lastSQL=“SELECT DISTINCT spdvc.tenant_id, spmar.user_group_id, spdvc.rule_id, spdvc.rule_comb_id, spdvc.comb_number, spdvc.dimension_id, spdvc.dimension_code, spdvc.column_name, spdvc.value_code, spdvc.no_value_flag, spdvc.auth_dimension_code, spdvc.hierarchy_scope, spdvc.permission FROM sys_perm_dim_value_cache spdvc, sys_perm_model_ass_rule spmar WHERE spdvc.rule_id = spmar.rule_id AND spdvc.rule_id = ? AND spdvc.rule_comb_id = ? AND spmar.model_id = ? AND EXISTS (SELECT 1 FROM sys_permission_rule r WHERE r.id = spdvc.rule_id AND r.rule_status = 1002 AND r.tenant_id = 1476472055858286593) AND spdvc.tenant_id = 1476472055858286593 [arguments: (1665921952692371458, 1665922630806470657, 1491426741154992130)]”] [err=“runtime error: invalid memory address or nil pointer dereference”]
TiDB logs are as follows:
2023-06-06T17:33:27.848080481+08:00 [2023/06/06 17:33:27.847 +08:00] [WARN] [server.go:644] [“Server.onConn handshake”] [conn=1669219179662111683] [error=“write tcp 172.18.65.182:4000->100.97.231.131:53009: write: connection reset by peer”] [“remote addr”=100.97.231.131:53009]

2023-06-06T17:33:28.040586807+08:00 [2023/06/06 17:33:28.040 +08:00] [ERROR] [pd.go:111] [“store min resolved ts not enabled, addr: http://dev-pd:2379”]

2023-06-06T17:33:28.040615362+08:00 [2023/06/06 17:33:28.040 +08:00] [ERROR] [pd.go:111] [“store min resolved ts not enabled, addr: http://dev-pd:2379”]

2023-06-06T17:33:28.040752317+08:00 [2023/06/06 17:33:28.040 +08:00] [ERROR] [pd.go:111] [“store min resolved ts not enabled, addr: http://dev-pd:2379”]

2023-06-06T17:33:28.124709497+08:00 [2023/06/06 17:33:28.124 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105561] [idle=15m0.000398889s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32812: i/o timeout”]

2023-06-06T17:33:28.233214818+08:00 [2023/06/06 17:33:28.233 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105563] [idle=15m0.000878951s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32816: i/o timeout”]

2023-06-06T17:33:28.543182969+08:00 [2023/06/06 17:33:28.543 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105565] [idle=15m0.000410374s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32822: i/o timeout”]

2023-06-06T17:33:28.642755832+08:00 [2023/06/06 17:33:28.642 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105569] [idle=15m0.000919631s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32824: i/o timeout”]

2023-06-06T17:33:28.843017215+08:00 [2023/06/06 17:33:28.842 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105571] [idle=15m0.000190512s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32828: i/o timeout”]

2023-06-06T17:33:29.173371810+08:00 [2023/06/06 17:33:29.173 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105575] [idle=15m0.000270542s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32836: i/o timeout”]

2023-06-06T17:33:29.273388896+08:00 [2023/06/06 17:33:29.273 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105577] [idle=15m0.000674472s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32838: i/o timeout”]

2023-06-06T17:33:29.367771865+08:00 [2023/06/06 17:33:29.367 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105579] [idle=15m0.000517187s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32840: i/o timeout”]

2023-06-06T17:33:29.374504283+08:00 [2023/06/06 17:33:29.374 +08:00] [INFO] [session.go:3846] [“CRUCIAL OPERATION”] [conn=1669219179662107119] [schemaVersion=93151] [“secure text”=“grant select,insert,update,delete,alter,drop on ms_test.auth_ypd_bp_01_fact003 to polaris@‘%’;”] [user=polaris@%]

2023-06-06T17:33:29.468915764+08:00 [2023/06/06 17:33:29.468 +08:00] [ERROR] [conn.go:1072] [“connection running loop panic”] [conn=1669219179662106373] [lastSQL=“SELECT DISTINCT spdvc.tenant_id, spmar.user_group_id, spdvc.rule_id, spdvc.rule_comb_id, spdvc.comb_number, spdvc.dimension_id, spdvc.dimension_code, spdvc.column_name, spdvc.value_code, spdvc.no_value_flag, spdvc.auth_dimension_code, spdvc.hierarchy_scope, spdvc.permission FROM sys_perm_dim_value_cache spdvc, sys_perm_model_ass_rule spmar WHERE spdvc.rule_id = spmar.rule_id AND spdvc.rule_id = ? AND spdvc.rule_comb_id = ? AND spmar.model_id = ? AND EXISTS (SELECT 1 FROM sys_permission_rule r WHERE r.id = spdvc.rule_id AND r.rule_status = 1002 AND r.tenant_id = 1476472055858286593) AND spdvc.tenant_id = 1476472055858286593 [arguments: (1665921952692371458, 1665922630806470657, 1491426741154992130)]”] [err=“runtime error: invalid memory address or nil pointer dereference”] [stack=“github.com/pingcap/tidb/server.(*clientConn).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1075\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:884\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:260\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:837\ngithub.com/pingcap/tidb/planner/core.getJoinHints\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/hints.go:128\ngithub.com/pingcap/tidb/planner/core.genHintsFromSingle\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/hints.go:242\ngithub.com/pingcap/tidb/planner/core.GenHintsFromFlatPlan\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/hints.go:47\ngithub.com/pingcap/tidb/executor.getEncodedPlan\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1700\ngithub.com/pingcap/tidb/executor.(*ExecStmt).SummaryStmt.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1757\ngithub.com/pingcap/tidb/util/stmtsummary.newStmtSummaryByDigestElement\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/stmtsummary/statement_summary.go:635\ngithub.com/pingcap/tidb/util/stmtsummary.(*stmtSummaryByDigest).add.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/stmtsummary/statement_summary.go:590\ngithub.com/pingcap/tidb/util/stmtsummary.(*stmtSummaryByDigest).add\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/stmtsummary/statement_summary.go:601\ngithub.com/pingcap/tidb/util/stmtsummary.(*stmtSummaryByDigestMap).AddStatement\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/stmtsummary/statement_summary.go:344\ngithub.com/pingcap/tidb/util/stmtsummary/v2.Add\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/stmtsummary/v2/stmtsummary.go:537\ngithub.com/pingcap/tidb/executor.(*ExecStmt).SummaryStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1855\ngithub.com/pingcap/tidb/executor.(*ExecStmt).FinishExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1342\ngithub.com/pingcap/tidb/executor.(*ExecStmt).CloseRecordSet\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1415\ngithub.com/pingcap/tidb/executor.(*recordSet).Close\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:180\ngithub.com/pingcap/tidb/session.(*execStmtResult).Close\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2425\ngithub.com/pingcap/tidb/server.(*tidbResultSet).Close\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:446\ngithub.com/pingcap/tidb/parser/terror.Call\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:315\ngithub.com/pingcap/tidb/server.(*clientConn).executePreparedStmtAndWriteResult\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn_stmt.go:357\ngithub.com/pingcap/tidb/server.(*clientConn).executePlanCacheStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn_stmt.go:226\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmtExecute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn_stmt.go:218\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1399\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1153\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:677”]

2023-06-06T17:33:29.685414503+08:00 [2023/06/06 17:33:29.685 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105585] [idle=15m0.000257125s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32846: i/o timeout”]

2023-06-06T17:33:29.715193621+08:00 [2023/06/06 17:33:29.715 +08:00] [WARN] [server.go:644] [“Server.onConn handshake”] [conn=1669219179662111689] [error=“write tcp 172.18.65.182:4000->100.116.37.129:25794: write: connection reset by peer”] [“remote addr”=100.116.37.129:25794]

2023-06-06T17:33:29.858878817+08:00 [2023/06/06 17:33:29.858 +08:00] [WARN] [server.go:644] [“Server.onConn handshake”] [conn=1669219179662111695] [error=“write tcp 172.18.65.182:4000->100.97.231.131:61144: write: connection reset by peer”] [“remote addr”=100.97.231.131:61144]

2023-06-06T17:33:29.999050454+08:00 [2023/06/06 17:33:29.998 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105587] [idle=15m0.000207793s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32852: i/o timeout”]

2023-06-06T17:33:30.040781611+08:00 [2023/06/06 17:33:30.040 +08:00] [ERROR] [pd.go:111] [“store min resolved ts not enabled, addr: http://dev-pd:2379”]

2023-06-06T17:33:30.040899891+08:00 [2023/06/06 17:33:30.040 +08:00] [ERROR] [pd.go:111] [“store min resolved ts not enabled, addr: http://dev-pd:2379”]

2023-06-06T17:33:30.040946285+08:00 [2023/06/06 17:33:30.040 +08:00] [ERROR] [pd.go:111] [“store min resolved ts not enabled, addr: http://dev-pd:2379”]

2023-06-06T17:33:30.225010730+08:00 [2023/06/06 17:33:30.224 +08:00] [INFO] [conn.go:1125] [“read packet timeout, close this connection”] [conn=1669219179662105593] [idle=15m0.000304069s] [waitTimeout=900] [error=“read tcp 172.18.65.182:4000->172.18.74.236:32856: i/o timeout”]

2023-06-06T17:33:30.424524120+08:00 [2023/06/06 17:33:30.

| username: xfworld | Original post link

What kind of node is 172.18.74.236?

| username: polars | Original post link

It is a relay server that performs port forwarding.

| username: xfworld | Original post link

The logs all show i/o timeout, you need to check it carefully.

| username: polars | Original post link

This is a relay service port for developers. These timeouts are likely due to the local development client timing out. The business cluster is directly connected and does not use this port, so the business errors are unrelated to this.

| username: xfworld | Original post link

I don’t know what scenario… :upside_down_face: Quite confused.

| username: Anna | Original post link

You can refer to this: runtime error: invalid memory address or nil pointer dereference · Issue #2109 · gruntwork-io/terragrunt · GitHub

| username: redgame | Original post link

I/O can be tested.

| username: polars | Original post link

This environment was fine with version 6.5 before and has been running for more than half a year, so there shouldn’t be any IO issues.

| username: yilong | Original post link

If you directly connect to the TiDB server to execute this SQL, will it report an error? Please test it. If it still reports an error, please also provide the table structure.

| username: polars | Original post link

Direct connection via SQL client can be executed successfully, the error only occasionally occurs on the application side.

| username: linnana | Original post link

If you identify the cause, please share it.

| username: Billmay表妹 | Original post link

You can join the group to consult with the official teachers~

| username: Billmay表妹 | Original post link

Based on the information you provided, it is not possible to determine the specific cause of the problem. However, based on the error message “connection running loop panic” and “connection closed,” it can be preliminarily judged that there was an exception in the TiDB connection, causing the application to be unable to access the TiDB database normally.

Possible reasons include:

  1. Network issues: There was an anomaly in the network connection between TiDB and the application, causing the connection to be closed.
  2. Resource issues: The TiDB cluster’s resources were insufficient, leading to the connection being closed.
  3. TiDB configuration issues: Improper TiDB configuration caused the connection to be closed.

To determine the specific cause of the problem, it is recommended that you check the logs of both TiDB and the application, as well as relevant monitoring information for TiDB, such as the TiDB Dashboard. You can analyze the problem based on the logs and monitoring information and take appropriate measures to resolve the issue.

Additionally, it is recommended to upgrade TiDB to the latest version for better performance and stability. You can also consider using tools officially provided by TiDB for performance tuning and troubleshooting, such as TiDB Binlog and TiDB Lightning.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.