TiDB Node Reaches Memory Limit Without Automatic Restart

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb节点到达内存上限不自动重启

| username: CHENGX

[TiDB Usage Environment] Production Environment
[TiDB Version] v5.4.3
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Issue Phenomenon and Impact] The memory of the TiDB node is set to 70G. Recently, it was found that the memory of the TiDB node gradually increased to 70G and got stuck, and it did not restart. After restarting, the issue still recurred. No abnormal SQL was found.
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]

| username: Kongdom | Original post link

Is it a mixed deployment? Have you checked the TiDB logs?

| username: tidb菜鸟一只 | Original post link

You can grep "expensive_query" in tidb.log, which will record SQL queries that run overtime or exceed the memory threshold. If the SQL cannot be optimized temporarily, try setting the tidb_mem_quota_query parameter.

| username: fish | Original post link

There are a lot of these errors in the TiDB logs:
[2023/09/05 04:43:16.303 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/scalar.go:77\ngithub.com/pingcap/tidb/statistics.(*Histogram).outOfRangeRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:896\ngithub.com/pingcap/tidb/statistics.(*Column).GetColumnRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:1247\ngithub.com/pingcap/tidb/statistics.(*HistColl).GetRowCountByColumnRanges\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:380\ngithub.com/pingcap/tidb/statistics.(*HistColl).Selectivity\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/selectivity.go:245\ngithub.com/pingcap/tidb/planner/core.(*DataSource).deriveStatsByFilter\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:250\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:406\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:131\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:448\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:284\ngithub.com/pingcap/tidb/planner.optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:385\ngithub.com/pingcap/tidb/planner.Optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:225\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/compiler.go:77\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1698\ngithub.com/pingcap/tidb/session.(*session).ExecRestrictedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1632\ngithub.com/pingcap/tidb/bindinfo.(*BindHandle).Update\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/bindinfo/handle.go:142\ngithub.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1110”]
[2023/09/05 04:43:19.303 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/scalar.go:77\ngithub.com/pingcap/tidb/statistics.(*Histogram).outOfRangeRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:896\ngithub.com/pingcap/tidb/statistics.(*Column).GetColumnRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:1247\ngithub.com/pingcap/tidb/statistics.(*HistColl).GetRowCountByColumnRanges\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:380\ngithub.com/pingcap/tidb/statistics.(*HistColl).Selectivity\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/selectivity.go:245\ngithub.com/pingcap/tidb/planner/core.(*DataSource).deriveStatsByFilter\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:250\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:406\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:131\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:448\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:284\ngithub.com/pingcap/tidb/planner.optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:385\ngithub.com/pingcap/tidb/planner.Optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:225\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/compiler.go:77\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1698\ngithub.com/pingcap/tidb/session.(*session).ExecRestrictedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1632\ngithub.com/pingcap/tidb/bindinfo.(*BindHandle).Update\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/bindinfo/handle.go:142\ngithub.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1110”]
[2023/09/05 04:43:22.304 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/scalar.go:77\ngithub.com/pingcap/tidb/statistics.(*Histogram).outOfRangeRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:896\ngithub.com/pingcap/tidb/statistics.(*Column).GetColumnRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:1247\ngithub.com/pingcap/tidb/statistics.(*HistColl).GetRowCountByColumnRanges\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:380\ngithub.com/pingcap/tidb/statistics.(*HistColl).Selectivity\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/selectivity.go:245\ngithub.com/pingcap/tidb/planner/core.(*DataSource).deriveStatsByFilter\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:250\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:406\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:131\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:448\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:284\ngithub.com/pingcap/tidb/planner.optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:385\ngithub.com/pingcap/tidb/planner.Optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:225\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/compiler.go:77\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1698\ngithub.com/pingcap/tidb/session.(*session).ExecRestrictedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1632\ngithub.com/pingcap/tidb/bindinfo.(*BindHandle).Update\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/bindinfo/handle.go:142\ngithub.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1110”]

| username: Kongdom | Original post link

Is there TiCDC?

| username: redgame | Original post link

Is this value present in the database?

| username: 大飞哥online | Original post link

No follow-up :rofl:

| username: CHENGX | Original post link

There isn’t any.