/tidb_data/prometheus-9090/docdb is taking up a lot of space, how to clean it up

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: /tidb_data/prometheus-9090/docdb 占用大量空间,如何清理

| username: EDG-给我冲

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.1.0
[Encountered Problem: Problem Phenomenon and Impact]
There are a large number of vlog data in the /tidb_data/prometheus-9090/docdb directory, occupying a lot of disk space. After checking related issues in the community, many people suggested scaling down and then scaling up, but do I have to scale down and up every few days? That’s too much. Others suggested configuring the parameter --storage.tsdb.retention=3d. I set it to 3 days and restarted Prometheus, but files older than three days still exist in the docdb. By checking the related process, it was found that the vlog files were generated by the ng-monitoring-server process, using the configuration file /tidb_deploy/prometheus-9090/conf/ngmonitoring.toml. In this configuration file, is it possible to configure the storage duration of related files, or is there any other way to limit the retention of these files?

| username: Fly-bird | Original post link

Here are the monitoring logs, try compressing and backing them up.

| username: EDG-给我冲 | Original post link

I don’t want to back up, just delete. I can’t just set up a cron job with rm, there should be a parameter for this, but I don’t know which one.

| username: tidb菜鸟一只 | Original post link

I see that my directory only has data from the last 4 days. Check the timestamps on your files.

| username: EDG-给我冲 | Original post link

The vlog files are from the 11th to today, a total of 5 days.

| username: Rilakkuma | Original post link

If the docdb files are not needed, you can directly remove them with rm.
Changing storage.tsdb.retention in the conf file might not take effect; you need to modify it in the template file at .tiup/storage/cluster/clusters/${clustername}/config-cache/run_prometheus_{ip}_{port}.sh and then restart Prometheus.

| username: dcswinner | Original post link

I encountered the same issue and just deleted it directly.

| username: liuis | Original post link

Modify the expiration policy.

| username: TiDB_C罗 | Original post link

I deleted some files, and the /tidb-deploy directory became smaller, but when I checked the disk where /tidb-deploy is located using df, it didn’t show any reduction.

| username: cassblanca | Original post link

rm -rf You deserve it

| username: Edwin | Original post link

ng-monitoring mainly includes two features: topsql and continuous profiling. Topsql data exceeding one month will be recycled by default, and this cannot be configured for now. Continuous profiling can be configured, with a default of 3 days:
Continuous profiling occupies relatively more space, and you can delete docdb to alleviate this (also, in version 6.1, conprof is enabled by default. If you delete docdb, the switch will reset to the default value. If you want to turn off conprof to save disk space, remember to check the dashboard to ensure the switch is off after deleting docdb).

| username: zhanggame1 | Original post link

You can delete it directly.

| username: redgame | Original post link

Listen to everyone, delete it.

| username: cassblanca | Original post link

rm -rf Simple and crude

| username: dba-kit | Original post link

The logs of Prometheus can be modified through storage_retention. You can refer to the documentation; actually, there are many customizable options. 通过 TiUP 部署 TiDB 集群的拓扑文件配置 | PingCAP 文档中心

| username: dba-kit | Original post link

Another major part is the logs generated by TopSQL and continuous profiling. You can refer to Edwin’s answer and make modifications through the dashboard page.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.