After TiKV scale-in, it remains in Tombstone state due to machine communication failure, causing the prune command to fail

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv scale-in 后处于 Tombstone 因机器无法通信,导致无法成功执行 prune 命令

| username: withseid

Background: Machine 37 is malfunctioning and cannot establish an SSH connection, so the TiKV node needs to be taken offline.

After executing the scale-in command, the TiKV node shows as Tombstone.

When executing the tiup cluster prune cluster_name command, the following error is reported. This error is also due to the inability to connect via SSH.

Error: failed to destroy tombstone: failed to stop tikv: failed to stop: 10.20.70.37 tikv-20160.service, please check the instance's log(/ssd/tidb-deploy/tikv-20160/log) for more detail.:executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@10.20.70.37:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin/usr/bin/sudo -H bash -c "systemctl daemon-reload && systemctl stop tikv-20160.service"}, cause: ssh: handshake failed: read tcp 10.20.70.39:33600->10.20.70.37:22: read: connection reset by peer

Verbose debug logs have been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2022-06-29-01-32-17.log.
Error: run `/home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster` (wd:/home/tidb/.tiup/data/TA6dw59) failed: exit status 1

Besides the prune command, what other methods can be used to take the TiKV node in Tombstone status offline?

| username: ddhe9527 | Original post link

Try this:

pd-ctl store remove-tombstone
| username: h5n1 | Original post link

tiup cluster prune --force
Then pd-ctl store remove-tombstone

| username: withseid | Original post link

It’s done, thank you.

| username: 我是咖啡哥 | Original post link

I have encountered this too, just add force and it will be fine. :blush:

| username: system | Original post link

This topic was automatically closed 1 minute after the last reply. No new replies are allowed.