TiDB Server Memory Anomaly Alert

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb服务器内存异常告警

| username: TiDBer_zWsmzAL8

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed that led to the issue
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]







The SQL is not running much, but the memory usage is not decreasing. I don’t know the reason.

| username: 像风一样的男子 | Original post link

Check the TopSQL monitoring.

| username: TiDBer_zWsmzAL8 | Original post link

The key point is that there aren’t many SQL queries running in the database, and they all execute very quickly.

| username: Billmay表妹 | Original post link

[Resource Allocation] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page.
Please take a screenshot to show your configuration.

| username: h5n1 | Original post link

Refer to this page to export the monitoring data of tidb \ tikv -detail \ overview \ node-exporter half an hour before and during the memory increase. Make sure to expand all panels and wait for all data to load before exporting, otherwise there will be no data.

https://metricstool.pingcap.net/#backup-with-dev-tools

| username: Billmay表妹 | Original post link

To supplement the background information:

Machines: Three 8-core 32GB ones

| username: 有猫万事足 | Original post link

I feel that during this period, there should be a lot of inserts in your database.

The TSO wait has increased, not just memory, but CPU should also have a significant rise. Looking at the locks and region misses, it seems like there are a lot of inserts.

| username: Billmay表妹 | Original post link

Please post the topsql.

| username: redgame | Original post link

Yes, this graph looks like a large amount of data is being written.

| username: ShawnYan | Original post link

TiDB Dashboard is a great tool, use it~

| username: 大飞哥online | Original post link

Slow SQL, take a look.
Your TiKV and TiDB are shared, check whether TiDB or TiKV is using more memory.