Deploy Monitoring Separately Using the TiUP Tool

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用tiup工具单独部署监控

| username: residentevil

[TiDB Usage Environment] Production Environment
[TiDB Version] V6.5.8
[Encountered Problem: Problem Description and Impact] Seeking advice from experts, the steps in 集群监控部署 | PingCAP 文档中心 seem quite numerous. Is it possible to use the tiup tool to deploy monitoring separately [outside of the TiDB cluster deployment period]?

| username: WalterWj | Original post link

Isn’t this a manual deployment? Deploying Prometheus with tiup is one cluster at a time.

| username: Jellybean | Original post link

If you want to use tiup to deploy monitoring, you still need to prepare the relevant monitoring packages, replace the ones provided by the official, and then deploy the replacement through tiup. For details, you can refer to:

| username: YuchongXU | Original post link

Download the TiDB-community-server package from the TiDB official website download page and extract it.

In the extracted files, find grafana-v{version}-linux-amd64.tar.gz and extract it.

| username: redgame | Original post link

When deploying a TiDB cluster using TiUP, TiUP will automatically deploy monitoring components such as Prometheus, Grafana, and Alertmanager, and automatically add monitoring configurations for new nodes during cluster expansion. The monitoring components automatically deployed by TiUP are not the latest versions of these third-party components. If you need to use the latest third-party components, you can upgrade the required monitoring components according to the methods described in this article.

| username: 小于同学 | Original post link

tiuph will automatically deploy these.

| username: residentevil | Original post link

I will take a closer look and verify this part. It seems like there are quite a few steps, haha.

| username: TiDBer_5cwU0ltE | Original post link

Is this required for actual work, or is it something you want to experiment with yourself?

| username: tidb菜鸟一只 | Original post link

I don’t understand your requirement. Do you want to add monitoring to an existing cluster that currently has no monitoring, or do you want to directly deploy monitoring from scratch?

| username: residentevil | Original post link

Add monitoring to a cluster without monitoring.

| username: TiDBer_aaO4sU46 | Original post link

Installing it separately indeed involves many steps.

| username: residentevil | Original post link

Is the node_exporter component installed by default when deploying the cluster, even if monitoring is not deployed?

| username: tidb菜鸟一只 | Original post link

If you don’t have a cluster, node_export won’t be automatically deployed. You really need to deploy it manually…

| username: residentevil | Original post link

When deploying the cluster, if Prometheus and Grafana components are not installed, is the node_exporter component also installed by default?

| username: tidb菜鸟一只 | Original post link

Yes, it will start node_exporter even if you don’t configure it by default.

| username: zhanggame1 | Original post link

Why deploy manually? If monitoring is not installed, just scale out. Monitoring can be scaled in and out.

| username: residentevil | Original post link

Are there detailed steps? Send them over for a look.

| username: zhanggame1 | Original post link

Test uninstall monitoring

| username: zhanggame1 | Original post link

Scaling Monitoring:
vi scale-out.yml

monitored:
 node_exporter_port: 9100
 blackbox_exporter_port: 9115

monitoring_servers:
 - host:  127.0.0.1

grafana_servers:
 - host:  127.0.0.1

Scaling out

root@tidb:~# tiup cluster scale-out tidb-test scale-out.yml -u root -p
tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.14.1/tiup-cluster scale-out tidb-test scale-out.yml -u root -p
You have one or more of ["global", "monitored", "server_configs"] fields configured in
        the scale out topology, but they will be ignored during the scaling out process.
        If you want to use configs different from the existing cluster, cancel now and
        set them in the specification fields for each host.
Do you want to continue? [y/N]: (default=N) y
Input SSH password:

+ Detect CPU Arch Name
  - Detecting node 127.0.0.1 Arch info ... Done

+ Detect CPU OS Name
  - Detecting node 127.0.0.1 OS info ... Done
Please confirm your topology:
Cluster type:    tidb
Cluster name:    tidb-test
Cluster version: v7.6.0
Role        Host       Ports       OS/Arch       Directories
----        ----       -----       -------       -----------
prometheus  127.0.0.1  9090/12020  linux/x86_64  /tidb-deploy/prometheus-9090,/tidb-data/prometheus-9090
grafana     127.0.0.1  3000        linux/x86_64  /tidb-deploy/grafana-3000
Attention:
    1. If the topology is not what you expected, check your yaml file.
    2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=127.0.0.1
+ [Parallel] - UserSSH: user=tidb, host=127.0.0.1
+ [Parallel] - UserSSH: user=tidb, host=127.0.0.1
+ Download TiDB components
  - Download prometheus:v7.6.0 (linux/amd64) ... Done
  - Download grafana:v7.6.0 (linux/amd64) ... Done
+ Initialize target host environments
+ Deploy TiDB instance
  - Deploy instance prometheus -> 127.0.0.1:9090 ... Done
  - Deploy instance grafana -> 127.0.0.1:3000 ... Done
+ Copy certificate to remote host
+ Generate scale-out config
  - Generate scale-out config prometheus -> 127.0.0.1:9090 ... Done
  - Generate scale-out config grafana -> 127.0.0.1:3000 ... Done
+ Init monitor config
Enabling component prometheus
        Enabling instance 127.0.0.1:9090
        Enable instance 127.0.0.1:9090 success
Enabling component grafana
        Enabling instance 127.0.0.1:3000
        Enable instance 127.0.0.1:3000 success
Enabling component node_exporter
        Enabling instance 127.0.0.1
        Enable 127.0.0.1 success
Enabling component blackbox_exporter
        Enabling instance 127.0.0.1
        Enable 127.0.0.1 success
+ [ Serial ] - Save meta
+ [ Serial ] - Start new instances
Starting component prometheus
        Starting instance 127.0.0.1:9090
        Start instance 127.0.0.1:9090 success
Starting component grafana
        Starting instance 127.0.0.1:3000
        Start instance 127.0.0.1:3000 success
Starting component node_exporter
        Starting instance 127.0.0.1
        Start 127.0.0.1 success
Starting component blackbox_exporter
        Starting instance 127.0.0.1
        Start 127.0.0.1 success
+ Refresh components configs
  - Generate config pd -> 127.0.0.1:2379 ... Done
  - Generate config tikv -> 127.0.0.1:20160 ... Done
  - Generate config tidb -> 127.0.0.1:4000 ... Done
  - Generate config prometheus -> 127.0.0.1:9090 ... Done
  - Generate config grafana -> 127.0.0.1:3000 ... Done
+ Reload prometheus and grafana
+ [ Serial ] - UpdateTopology: cluster=tidb-test
Scaled cluster `tidb-test` out successfully

| username: residentevil | Original post link

The steps are very detailed, saving it, haha :+1: