Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: TiDB V5 encountered error报错后 tidb-server OOM
|
username: TiDBer_yyy
[Test Environment for TiDB] Testing
[TiDB Version] 5.0.4
[Reproduction Path] Around 18:30 on January 28th, tidb-server OOM; multiple OOMs between 09:54-10:03 on January 29th
[Encountered Problem: Phenomenon and Impact]
Multiple occurrences of: [terror.go:291] [“encountered error”] followed by tidb-server OOM within 1 minute
[2023/01/28 18:26:30.792 +08:00] [ERROR] [terror.go:291] ["encountered error"] [error="[types:1292]Incorrect time value: '{0 0 0 0 0 0 0}'"] [stack="github.com/pingcap/parser/terror.Log\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20210623034316-5ee95ed0081f/terror/terror.go:291\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:66\ngithub.com/pingcap/tidb/statistics.(*Histogram).PreCalculateScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:93\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initStatsBuckets\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:369\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).InitStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:406\ngithub.com/pingcap/tidb/domain.(*Domain).loadStatsWorker\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/domain/domain.go:1177"]
[2023/01/28 18:26:30.792 +08:00] [ERROR] [terror.go:291] ["encountered error"] [error="[types:1292]Incorrect time value: '{0 0 0 0 0 0 0}'"] [stack="github.com/pingcap/parser/terror.Log\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20210623034316-5ee95ed0081f/terror/terror.go:291\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:66\ngithub.com/pingcap/tidb/statistics.(*Histogram).PreCalculateScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:94\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initStatsBuckets\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:369\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).InitStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:406\ngithub.com/pingcap/tidb/domain.(*Domain).loadStatsWorker\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/domain/domain.go:1177"]
[2023/01/28 18:26:30.953 +08:00] [INFO] [domain.go:1181] ["init stats info time"] ["take time"=15.752796356s]
[2023/01/28 18:26:31.265 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1907] ["cost time"=49.349023ms]
[2023/01/28 18:26:31.756 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=73.369494ms]
[2023/01/28 18:26:32.601 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=34.266625ms]
[2023/01/28 18:26:33.310 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1907] ["cost time"=40.109161ms]
[2023/01/28 18:26:35.282 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1907] ["cost time"=50.723493ms]
[2023/01/28 18:26:35.329 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=76.530381ms]
[2023/01/28 18:26:36.450 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=30.648492ms]
[2023/01/28 18:26:37.240 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=37.580602ms]
[2023/01/28 18:26:39.114 +08:00] [WARN] [memory_usage_alarm.go:141] ["tidb-server has the risk of OOM. Running SQLs and heap profile will be recorded in record path"] ["is server-memory-quota set"=false] ["system memory total"=33566154752] ["system memory usage"=26890240000] ["tidb-server memory usage"=8267442624] [memory-usage-alarm-ratio=0.8] ["record path"="/tmp/3013_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record"]
[2023/01/28 18:26:39.135 +08:00] [INFO] [manager.go:325] ["get owner"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager da53c763-89c4-47b6-bf64-f4328edd517b"] [ownerID=da53c763-89c4-47b6-bf64-f4328edd517b]
Question
How to find Running SQLs and heap profile
|
username: