Prometheus Data Cleanup

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: prometheus 清数据

| username: 等一分钟

How to clean up the historical data of TiDB’s Prometheus?

| username: 等一分钟 | Original post link

]# ll /data/tidb-data/prometheus-8249/
total 96
drwxr-xr-x  3 tidb tidb 4096 Feb  9 11:02 01GRT1Z9V470R228801SSCV1VP.tmp
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSC2SPKY3M5RZVA41ND8SVT6
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSHW663289J8Y1M2MN3HV4AV
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSKSY6B8QBWCRM07YW75PCPF
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSNQQPBM8Y10X1CKY6TS9YSC
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSQNGZ8G8VJ23XV8B5F86RH1
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSSKAZTC2FJFN2Z9F3HP0PP3
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSVH41B2WA7D69N7FR4YAZX3
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSXEXKK3NH1GFBZD856XCH54
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GSZCQ8FA57AGDM1GG0HNB8AD
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GT1AGT1JF27M0SGEYN3F32MY
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GT38A6RYWFC0P0N2KZ9BKV72
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GT563ZNR0JDJN5BAXPRMZXQ8
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GT73XEQDDPC4JZH22K0SMHH7
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GT91QDY6MBBAWSTGBR0XRP3D
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:15 01GTAZHJ1FN4BSBXNFX5S94XXA
drwxr-xr-x  3 tidb tidb 4096 Mar  1 11:14 01GTDJMZ2V892TSHYDEHP6CDZR.tmp
drwxr-xr-x  3 tidb tidb 4096 Mar  1 13:00 01GTDRN950281E94W98BY4NBBK
drwxr-xr-x  3 tidb tidb 4096 Mar  1 15:00 01GTDZH0CZYTVJQP4S9KD81BPB
drwxr-xr-x  3 tidb tidb 4096 Mar  1 15:07 01GTDZMFYS2BN5QKC2R9V22757
drwxr-xr-x  3 tidb tidb 4096 Mar  1 17:00 01GTE6CQN0AJ7XDGRF22Q1YJFS
drwxr-xr-x  3 tidb tidb 4096 Mar  1 19:00 01GTED8EX0WP1F0MB1CSGXBA5P
drwxr-xr-x  3 tidb tidb 4096 Mar  1 19:01 01GTEDBK9K805ZVW2FX1829BKZ.tmp
-rw-r--r--  1 tidb tidb    0 Feb 17  2022 lock
drwxr-xr-x 12 tidb tidb 4096 Mar  1 19:01 wal
| username: 等一分钟 | Original post link

Can I delete the file directly?

| username: liuis | Original post link

Doesn’t Prometheus have an automatic expiration policy?

| username: 等一分钟 | Original post link

Do I need to configure it, or is it available by default?

| username: DBRE | Original post link

You can use the Prometheus API to delete:

POST /api/v1/admin/tsdb/delete_series

Example:

$ curl -X POST \
  -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=up&match[]=process_start_time_seconds{job="prometheus"}'

Reference: HTTP API | Prometheus

| username: 等一分钟 | Original post link

[root@izuf6fv492k5e39tbrz23uz data]# du -sh tidb-data/prometheus-8249/
71G tidb-data/prometheus-8249/
[root@izuf6fv492k5e39tbrz23uz data]# du -sh data_dir/prometheus-11249/
105G data_dir/prometheus-11249/

| username: liuis | Original post link

There is a reply above, there is an API that can delete, and at the same time, when Prometheus starts, you can configure the expiration policy.

| username: 等一分钟 | Original post link

Why do I have two directories here?

| username: 等一分钟 | Original post link

It was started using tiup. Should the expiration policy be written in the configuration file?

| username: liuis | Original post link

Refer to the configuration in this document: 通过 TiUP 部署 TiDB 集群的拓扑文件配置 | PingCAP 文档中心 and set storage_retention.

| username: 等一分钟 | Original post link

Okay, let me take a look.

| username: TiDBer_pkQ5q1l0 | Original post link

Find the running run_prometheus.sh script, modify the storage.tsdb.retention at the bottom to the number of days you want, and restart. It will automatically clean up some historical data.

| username: liuis | Original post link

Others used tiup to start it.

| username: TiDBer_pkQ5q1l0 | Original post link

Using tiup will also call this script.


| username: Kongdom | Original post link

| username: liuis | Original post link

Alright, the official documentation has an introduction https://docs.pingcap.com/zh/tidb/stable/tiup-cluster-topology-reference#monitoring_servers. Refer to this for configuration and set storage_retention.

| username: 等一分钟 | Original post link

Alright, thanks everyone.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.