No data in Prometheus monitoring for tidb-lightning configuration

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb-lightning配置prometheus监控没有数据

| username: starCrush

【TiDB Usage Environment】Testing
【TiDB Version】
【Reproduction Path】What operations were performed to encounter the issue
【Encountered Issue: Problem Phenomenon and Impact】
After configuring Prometheus for tidb-lightning, there is no data in the monitoring seen on Grafana.
This is the lightning configuration:

This is the configuration added in prometheus.yml:

Can you help check if there is any misconfiguration?
【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】

The data on Grafana monitoring is all empty:

| username: dba-kit | Original post link

You wrote pprof-port, try changing it to status-addr.

# HTTP port for progress display web interface, fetching Prometheus monitoring items, exposing debugging data, and submitting import tasks (in server mode). Set to 0 to disable.
status-addr = ':8289'
| username: dba-kit | Original post link

For detailed configuration file instructions, see: TiDB Lightning 配置参数 | PingCAP 文档中心

| username: starCrush | Original post link

Isn’t this configuration incorrect?

| username: dba-kit | Original post link

I estimate that it is a very old configuration file, and the documentation has not been updated. Follow the link I posted, I have used status-addr, and you can see the web interface.

| username: starCrush | Original post link

Even with this configuration, I still can’t see the progress monitoring on Grafana.

| username: starCrush | Original post link

Additionally, from the logs, it appears that the import speed is gradually slowing down. Initially, it was over 200 MiB/s, but now it has decreased to just over 10 MiB/s (for about 10 TB of data).

| username: Billmay表妹 | Original post link

It is recommended to create a separate post to describe the issue of data import.

| username: Billmay表妹 | Original post link

First, ensure that you have correctly configured TiDB Lightning’s Prometheus monitoring according to the official documentation. If you have configured Prometheus monitoring according to the official documentation but see no data in Grafana, it could be due to one of the following reasons:

  1. Check if Prometheus is collecting monitoring data from TiDB Lightning. You can check in Prometheus’s Web UI to see if there is monitoring data from TiDB Lightning. If not, check if Prometheus’s configuration file correctly specifies TiDB Lightning’s monitoring address.

  2. Check if Grafana is correctly connected to Prometheus. You can check in Grafana’s data source settings to see if Prometheus’s address and port are correctly configured. If you are using TiDB Lightning’s default port, then Prometheus’s address should be http://<tidb-lightning-ip>:9090.

  3. Check if Grafana’s monitoring panel is correctly configured with Prometheus’s query statements. You can check in Grafana’s monitoring panel settings to see if the query statements correctly specify TiDB Lightning’s monitoring metrics. For example, if you want to view TiDB Lightning’s import speed, the query statement should be tidb_lightning_importer_speed_bytes.

If you have checked the above three aspects but still cannot see TiDB Lightning’s monitoring data in Grafana, try restarting Prometheus and Grafana and check the log files for more information. If the issue persists, try seeking help in the TiDB community forum.

| username: dba-kit | Original post link

In the beginning, it was over 200 MiB/s, but now it’s only around 10 MiB/s.

In this situation, it’s generally caused by encountering many small files. What tool are you using to export?

| username: starCrush | Original post link

The export is done using dumpling:
nohup /root/tidb/tidb-community-toolkit-v6.5.1-linux-amd64/dumpling -u my_dumpling -p ‘’ -P 3306 -h xxxxx --filetype sql --read-timeout=1h -t 16 -o /data3/tmp1 -r 200000 -F256MiB > /data3/dumpling.log 2>&1 &

| username: dba-kit | Original post link

Using -r does indeed cause this issue. You can check the dumpling documentation. When dumping TiDB data, the value after -r is invalid. No matter what value you set, it will use the chunk table inside TiDB to get the shard boundaries. If it’s very slow, you can check the logs to see which table is being written to and filter out that table first. Then, export that table separately without using -r.

| username: dba-kit | Original post link

Alternatively, and more simply, you can remove the -r option and only set the -F option. However, in this case, you won’t be able to perform concurrent exports on a single table, which will increase the export time for large tables.

| username: starCrush | Original post link

I am currently dumping MySQL data. Is this parameter also invalid? The current scenario involves some databases around 10TB that need to be migrated to TiDB. The individual tables are not large, just a few GB. Could you please provide a template for the dumpling export command?

| username: dba-kit | Original post link

The -r option is still effective for MySQL, as it is designed to export a single table with only one thread, so there shouldn’t be cases where some files are particularly small. Are you importing through logical import rather than directly importing into TiKV in local mode? :thinking:
If you are converting logically to SQL for import, it is indeed normal for larger tables to insert more slowly. You can check this out: 物理导入模式 | PingCAP 文档中心

| username: starCrush | Original post link

I am using the local mode, but it’s still slow, and there are quite a few issues: For example, after importing using the local mode, I need to analyze the table. For a table with less than a million rows, it takes more than 2 hours to complete the analysis. I have also enabled fast analyze, but it doesn’t seem to help (tidb5.4.3).

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.