This morning, I suddenly found that the database could not be connected. Checking tidb-server showed the message: Got too many pings from the client, closing the connection

translator_bot · June 22, 2024, 7:55pm

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 今天早上突然发现数据库连接不上了，查看tidb-server提示Got too many pings from the client, closing the connection.

| username: yinyuncan

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.3.0
[Reproduction Path] What operations were performed when the issue occurred
No operations were performed, the issue occurred suddenly
[Encountered Issue: Issue Phenomenon and Impact]
This is the error reported when the client connects

This is the error reported by the Java program

### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20, creating 0, createErrorCount 272
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
        ... 51 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20, creating 0, createErrorCount 272
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:83)
        at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
        at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67)
        at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337)
        at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:86)
        at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
        at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:81)
        at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62)
        at jdk.proxy2/jdk.proxy2.$Proxy385.query(Unknown Source)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151)
        ... 58 common frames omitted
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20, creating 0, createErrorCount 272
        at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1775)
        at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1429)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5059)
        at com.alibaba.druid.filter.FilterAdapter.dataSource_getConnection(FilterAdapter.java:2756)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5055)
        at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:726)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5055)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1407)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1399)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:100)
        at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:159)
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:117)
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
        ... 69 common frames omitted
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
        at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
        at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
        at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
        at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
        at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:156)
        at com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:787)
        at com.alibaba.druid.filter.FilterEventAdapter.connection_connect(FilterEventAdapter.java:38)
        at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
        at com.alibaba.druid.filter.stat.StatFilter.connection_connect(StatFilter.java:251)
        at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
        at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1669)
        at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1733)
        at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2840)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at jdk.internal.reflect.GeneratedConstructorAccessor85.newInstance(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
        at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:544)
        at com.mysql.cj.protocol.a.NativeProtocol.readServerCapabilities(NativeProtocol.java:496)
        at com.mysql.cj.protocol.a.NativeProtocol.beforeHandshake(NativeProtocol.java:383)
        at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1346)
        at com.mysql.cj.NativeSession.connect(NativeSession.java:157)
        at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:956)
        at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:826)
        ... 12 common frames omitted
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:538)
        ... 18 common frames omitted

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

This is the error log data

[2023/01/10 08:44:39.402 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:45:39.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:45:39.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:46:49.400 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:46:49.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:47:39.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:47:39.402 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:48:39.405 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:48:39.410 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]

This is the server

translator_bot · June 22, 2024, 7:55pm

| username: yinyuncan | Original post link

This is the cluster inspection.

translator_bot · June 22, 2024, 7:55pm

| username: yinyuncan | Original post link

This issue feels completely baffling, but I’m not going to worry about it for now. I’m currently performing an upgrade operation, upgrading to version 6.5 first.

translator_bot · June 22, 2024, 7:55pm

| username: ealam_小羽 | Original post link

Reference this?

translator_bot · June 22, 2024, 7:55pm

| username: ffeenn | Original post link

Please provide the TiDB Overview monitoring panel for review.

translator_bot · June 22, 2024, 7:55pm

| username: yinyuncan | Original post link

This issue has been resolved, and it doesn’t seem to be a TiDB problem, just a misunderstanding

After the upgrade, I just tested that the intranet can connect.

Because I used APISIX for TCP load balancing earlier, I did update the APISIX configuration yesterday, which might have caused the issue.

I will continue to investigate the APISIX problem. Thanks for everyone’s help.

I haven’t seen any ERROR logs after the upgrade, which feels great.

I love TiDB.

translator_bot · June 22, 2024, 7:55pm

| username: ohammer | Original post link

When I encounter errors with the Navicat client, I usually try accessing it directly on the server using the shell.

translator_bot · June 22, 2024, 7:55pm

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.