How to Use a Single Prometheus for Multiple TiDB Dashboards

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何多个tidb- dashboard使用同一个prometheus

| username: Hacker_ynbNppAC

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.1
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration] 5 TiKV nodes, 3 PD nodes, 2 TiDB nodes
[Encountered Problem: Problem Phenomenon and Impact] How to use a single Prometheus for multiple TiDB dashboards

| username: ShawnYan | Original post link

Multiple TiDB dashboards? Does that mean multiple TiDB clusters? Sharing one Prometheus?

| username: Billmay表妹 | Original post link

Warning, you have already posted multiple spam posts today. Please do not maliciously farm points.

| username: Billmay表妹 | Original post link

If you want to contribute to the community, you can check out the unresolved issues.
Please do not post irrelevant content~
List of unresolved issues: TiDB 的问答社区 - TiDB 社区
Please try to provide good problem-solving ideas and reference suggestions, etc.
TiDBers in the TiDB community do not like spamming behavior~

If spamming is found again, the account will be banned for 7 days.

| username: Hacker_ynbNppAC | Original post link

Yes, we have many TiDB instances, and it feels wasteful to have one Prometheus for each instance.

| username: ShawnYan | Original post link

Setting up a separate Prometheus instance and connecting multiple TiDB clusters to it requires network connectivity, which poses security risks. Additionally, the Prometheus instance might experience high load, and what if there is a delay in alerting?

If it’s a testing environment, you can try configuring the IPs of all the nodes you want to monitor into that unified Prometheus instance and let it scan all the nodes.

| username: wzf0072 | Original post link

Refer to the manual deployment monitoring: 集群监控部署 | PingCAP 文档中心

| username: Hacker_ynbNppAC | Original post link

But in that case, the dashboard cannot be used.

| username: ShawnYan | Original post link

TiDB Dashboard and PD nodes are two different things from Prometheus.

| username: liuis | Original post link

The federated model is feasible.

| username: Hacker_ynbNppAC | Original post link

However, the TiDB dashboard needs to connect to PD. I know how to collect TiDB data and configure alerts. But in that case, the dashboard won’t be able to connect, and the interface will show an error like Necessary component NgMonitoring not started in the cluster. I also know that adding a configuration would solve this. But we are more concerned about costs, and having more than 30 Prometheus instances is too wasteful.

| username: Hacker_ynbNppAC | Original post link

However, the federated mode still cannot use the TiDB dashboard and will report an error. The interface prompts Necessary component NgMonitoring not started in the cluster.

| username: srstack | Original post link

ngm is deployed together with prom, so currently, it is difficult for TiUP to use a single prom for multiple clusters while ensuring complete functionality. You can look forward to TiUP using the vm cluster as a monitoring storage solution in the future. Of course, you can also research it yourself.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.