Things to Do After TiDB Deployment for Stable Operation

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb部署完为了稳定运行需要做的一些事情

| username: tidb狂热爱好者

  1. Configure Daily Backup:

To ensure data safety, it is very important to perform data backups in TiDB. You can use TiDB’s built-in backup tools tidb-lightning and br, or other third-party backup tools. The steps to configure daily backups are as follows:

  • Install and configure the backup tool
  • Write the backup script
  • Set up a scheduled task
  1. Configure TiDB Monitoring with DingTalk Alerts:

By monitoring the status of the TiDB cluster, you can promptly identify and resolve issues. TiDB supports the Grafana monitoring system and can send alert information to DingTalk groups through Grafana’s Alertmanager component. The steps to configure TiDB monitoring with DingTalk alerts are as follows:

  • Install Grafana and Alertmanager
  • Configure Alertmanager to send alert information to DingTalk
  • Configure Grafana monitoring dashboard and set alert rules
  1. Configure Case Sensitivity:

In some cases, you need to configure the TiDB cluster to be case-insensitive. You can achieve this by modifying the configuration file. The specific steps are as follows:

  • Open the TiDB configuration file (/path/to/tidb-server/conf/tidb.toml)
  • Add the following configuration item: [server] lowercase-table-names=true
  • Restart the TiDB cluster
  1. Enable Maximum SQL Execution Time Limit in TiDB:

In some cases, you need to limit the execution time of a single SQL to avoid affecting the performance of the entire cluster. You can achieve this by setting TiDB’s max-execution-time parameter. The specific steps are as follows:

  • Open the TiDB configuration file (/path/to/tidb-server/conf/tidb.toml)
  • Add the following configuration item: [performance] max-execution-time=10000 (unit is milliseconds, this example limits the execution time of a single SQL to 10 seconds)
  • Restart the TiDB cluster
| username: Kongdom | Original post link

It is recommended to also back up tiup. Behind every suggestion, there is an incredible past~ :rofl:

| username: 海石花47 | Original post link

May I ask, how do you send alerts with TiDB’s built-in monitoring? I don’t know how to do it.

| username: 海石花47 | Original post link

Please tell your story… tiup seems to be in the ~/.tiup/bin/tiup directory of the tidb user… is this normal? I don’t quite understand.

| username: Kongdom | Original post link

It is because the central control machine might have crashed. If there is no backup of tiup when rebuilding the central control machine, the cluster cannot be managed.

| username: 海石花47 | Original post link

Can’t this be reinstalled with tiup? My machine is an ECS server… there shouldn’t be this problem, right?

| username: 海石花47 | Original post link

So for my ~/.tiup/bin/tiup deployment (following the official documentation), should I directly back up the ~/.tiup directory?

| username: Kongdom | Original post link

Yes, you can directly back up this.

| username: BraveChen | Original post link

Haha :stuck_out_tongue_winking_eye: :grinning:

| username: BraveChen | Original post link

| username: ffeenn | Original post link

Do you perform backup data verification in your production environment besides verifying after the backup is completed? Here we also do verification for incremental backups.

| username: 人如其名 | Original post link

If you set it this way, it will be overwritten after tiup reload. You need to make changes in tiup cluster edit.

| username: 海石花47 | Original post link

Yes, it should be the dumpling tool, he probably wrote it wrong? Lighting is used for data recovery, right?

| username: 海石花47 | Original post link

How do you perform incremental backups? I specify the database tables and do a complete dump with Dumpling once a day, then compress and encrypt it before uploading to OSS for offsite storage.

| username: BraveChen | Original post link

That’s it, it’s usually like this.

| username: 裤衩儿飞上天 | Original post link

Incremental backup is not supported by br dumpling.

| username: Kongdom | Original post link

With clusters in use, backups are basically unnecessary. :grinning:

| username: 庙小妖风大 | Original post link

I copy the /root/.tiup directory in full (rsync) to the backup server every day.

| username: Kongdom | Original post link

:handshake: :handshake: :handshake: Like-minded

| username: 海石花47 | Original post link

By the way, does the tiup directory change frequently? Why do you need to back it up every day?