PD|UI Node Status Abnormal

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: PD|UI节点状态异常

| username: Kongdom

[TiDB Usage Environment] Production Environment
[TiDB Version] v5.4.3
[Reproduction Path] Forced server restart
[Encountered Problem: Issue Phenomenon and Impact]
After the server was forcibly restarted, the PD|UI node status changed to down, and the error logs are as follows. The cluster is operating normally, but the dashboard site cannot be opened. In this situation, should I perform a PD recover or scale in and out the abnormal node?

[2023/03/09 18:03:53.326 +08:00] [ERROR] [client.go:171] ["region sync with leader meet error"] [error="[PD:grpc:ErrGRPCRecv]rpc error: code = Unavailable desc = server not started"]
[2023/03/10 00:59:08.601 +08:00] [ERROR] [etcdutil.go:71] ["failed to get cluster from remote"] [error="[PD:etcd:ErrEtcdGetCluster]could not retrieve cluster information from the given URLs"]
[2023/03/10 00:59:18.603 +08:00] [ERROR] [etcdutil.go:71] ["failed to get cluster from remote"] [error="[PD:etcd:ErrEtcdGetCluster]could not retrieve cluster information from the given URLs"]
[2023/03/10 01:12:25.664 +08:00] [ERROR] [etcdutil.go:71] ["failed to get cluster from remote"] [error="[PD:etcd:ErrEtcdGetCluster]could not retrieve cluster information from the given URLs"]
[2023/03/10 01:12:25.684 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:12:40.945 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:12:56.447 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:13:11.950 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:13:27.372 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:13:42.876 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: tidb菜鸟一只 | Original post link

The UI node is the dashboard node, and the L node is the main node. If the main node is fine and only the UI node is down, you can directly scale in and out.

| username: 啦啦啦啦啦 | Original post link

Try using tiup to move the dashboard.

| username: Kongdom | Original post link

Oh, can it be switched directly? How do you do that?

| username: 啦啦啦啦啦 | Original post link

Refer to this, I have tested it in the test environment and it can be switched.

| username: Kongdom | Original post link

:+1: :+1: :+1: Yet another small detail~

| username: Kongdom | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.