Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: pd的leader找不到，导致集群不可用
[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
In the architecture of juicefs+tikv, during daily task runs, the PD leader could not be found, causing the cluster to be unavailable.
[Encountered Issue: Problem Phenomenon and Impact]
PD log error: load from etcd meet error, the first error occurred at 11:29, followed by a period of memory and coroutine surge, and finally, it could not recover automatically and had to manually restart the cluster.
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
This has happened many times in the quasi-production environment.
Since we saw that v7.1.0 updated this bug, we upgraded to this version. Now it seems that this cannot solve our problem.
Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page.
Thank you for your reply, cousin. However, we are currently unable to log in to the dashboard due to security issues and need to apply for access. Which specific monitoring graph would you like to see? I can take a screenshot from Grafana for you.
The logs show etcd read/write timeout. Check if the machine’s CPU or hard disk is overwhelmed. The subsequent memory and goroutine surge might be caused by a cascading failure due to the earlier timeout.
These are some other logs, namely pd.log and pd_stderr.log. We will start continuous analysis and provide feedback if we capture anything further.
You can check it this way as well. It won’t be a mixed deployment, right?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
|
username: 中国电信TIKV
|
username: 
|
username: 
|
username: 
|
username: