Failed to Start node_exporter-9100.service

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: node_exporter-9100.service服务启动失败

| username: Kongdom

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.3
[Reproduction Path] The cluster has undergone data center migration, PD reconstruction, expansion, and contraction.
[Encountered Problem: Problem Phenomenon and Impact]
node_exporter-9100 reload error
The error content is as follows:

|2023-06-29T05:46:24.958+0800|DEBUG|retry error|{error: operation timed out after 2m0s}|
|---|---|---|---|
|2023-06-29T05:46:24.958+0800|DEBUG|TaskFinish|{task: StartCluster, error: failed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.: timed out waiting for port 9100 to be started after 2m0s, errorVerbose: timed out waiting for port 9100 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:121\ngithub.com/pingcap/tiup/pkg/cluster/operation.systemctlMonitor.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:336\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594\nfailed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.}|
|2023-06-29T05:46:24.958+0800|INFO|Execute command finished|{code: 1, error: failed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.: timed out waiting for port 9100 to be started after 2m0s, errorVerbose: timed out waiting for port 9100 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:121\ngithub.com/pingcap/tiup/pkg/cluster/operation.systemctlMonitor.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:336\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594\nfailed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.}|

Checked the node_exporter.log log under the corresponding node, there are no records.

| username: 像风一样的男子 | Original post link

Check if the node_exporter process is running. If it is not, manually execute the startup script tidb-deploy/monitored-9100/scripts/run_node_exporter.sh to see if any logs are generated.

| username: Kongdom | Original post link

It should be normal.

| username: 像风一样的男子 | Original post link

Normally, it should be like this. Your status is “auto-restarting.”

| username: Kongdom | Original post link

:astonished: Try manually shutting it down at noon.

| username: xingzhenxiang | Original post link

I usually do it like this:
cd /etc/systemd/system/

systemctl start node_exporter-9100.service

systemctl start blackbox_exporter-9115.service

| username: Kongdom | Original post link

:+1: I’ll give it a try at noon.

| username: Kongdom | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.