After installing 3 TiKV components on a single node for testing, forcibly scaling down 2 of them causes TiDB to fail to start

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 单节点测试安装3个tikv组件后,强制缩容其中2个tikv,导致tidb启动失败

| username: Ti青涩

[TiDB Usage Environment] Testing
[TiDB Version] 6.1.1
[Reproduction Path] After installing 3 TiKV components on a single node for testing, forcibly scale down 2 of the TiKV components, causing TiDB to fail to start.
[Encountered Problem: Symptoms and Impact] According to the TiDB logs, it keeps trying to connect to the 2 TiKV components that have been taken offline.
[Resource Configuration] The test environment configuration is very low.
[Attachments: Screenshots/Logs/Monitoring]
Status of each component:


TiDB component logs:

TiKV component logs:

| username: Ti青涩 | Original post link

Is there any way to prevent TiDB from accessing those two TiKV nodes?

| username: 啦啦啦啦啦 | Original post link

It’s normal that it can’t start. With 3 replicas and 3 TiKV nodes, it can only tolerate 1 TiKV failure at most, which is equivalent to losing 2 replicas.

| username: Ti青涩 | Original post link

But I only have one node here, is that not possible?

| username: 啦啦啦啦啦 | Original post link

Definitely not, most of the replicas are gone. Refer to this link to try expanding and then repairing.

| username: Billmay表妹 | Original post link

Multiple Replica Loss Issue - Column Articles

Multiple Replica Loss Issue - Technical Q&A

Multiple Replica Loss Issue - SOP

SOP Series 18: TiUP Environment Recovery TiKV Replica - :milky_way: Operations Guide / TiDB Operations Manual - TiDB Q&A Community

For reference~

| username: tidb菜鸟一只 | Original post link

With multiple replicas in TiKV, if you directly take down 2 out of 3, it definitely won’t work.

| username: Ti青涩 | Original post link

I tried the solutions provided by everyone, but none of them worked because my scaling-down operation was too extreme. Since I had previously tested installing a single TiKV component, I thought it would be fine to scale down two of the three TiKV components, but I overlooked the default three-replica setting. I still want to thank everyone for helping me, a novice. Thank you very much. In the end, I reinstalled everything. When I have time, I will review other posts and experiment with those scenarios.

| username: tidb狂热爱好者 | Original post link

Yes, three nodes are the minimum option.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.