How TiKV Handles Split-Brain Scenarios

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiKV如何处理脑裂的情况

| username: alfred

To improve efficiency, please provide the following information. Clear problem descriptions can lead to quicker resolutions:

【Overview】 Scenario + Problem Overview

【Application Framework and Business Logic Adaptation】

【Background】 Actions Taken

【Phenomenon】 Business and Database Phenomenon

【Problem】 Current Issue

【Business Impact】

【TiDB Version】

【Attachments】 Relevant Logs and Monitoring (https://metricstool.pingcap.com/)

During the election time, even if the region Leader itself fails (e.g., due to a network partition), it still acts as the region Leader. At this time, data reading is done via Lease read, i.e., local read. So, will data writes to this region Leader still succeed? How does it coordinate with other region nodes?


For questions related to performance optimization or troubleshooting, please download and run the script. Be sure to select all and copy-paste the terminal output results for upload.

| username: 啦啦啦啦啦 | Original post link

You can learn about the Raft algorithm here:

| username: 突破边界 | Original post link

It won’t succeed because log replication won’t receive responses from the majority of nodes.

| username: alfred | Original post link

This is guaranteed by the Raft protocol.

| username: xfworld | Original post link

As long as the number of nodes is odd, it will be fine. If there is an even number of nodes, there may be incomplete replicas, which can cause the cluster to malfunction.

| username: TiDBer_rvITcue9 | Original post link

Raft protocol

| username: 饭光小团 | Original post link

The Raft protocol’s majority voting mechanism addresses the issue of network partitioning.

| username: 烂番薯0 | Original post link

Multiple replicas, along with a voting mechanism, where the leader will vote.

| username: DBAER | Original post link

For writes, the log must be accepted by the majority of nodes before it can return a successful write.

| username: Soysauce520 | Original post link

It is also necessary to pay attention to which PD and TiKV are in a network partition.

| username: 这里介绍不了我 | Original post link

Mainly ensured by Raft.

| username: YuchongXU | Original post link

Raft leader election

| username: chris-zhang | Original post link

You can first learn about the Raft protocol.

| username: Hacker_QGgM2nks | Original post link

TiKV handles split-brain scenarios primarily by relying on the Raft protocol. During the election time, even if the region Leader itself fails (such as a network partition), it still remains the region Leader. At this time, data is read using Lease read, i.e., local read. This method ensures linear consistency, meaning that if we write a value at a certain point in time, any read after that point will definitely read that value and not any value from before that point [*].

Additionally, if a split-brain occurs and the client requests reach the minority cluster, it will not receive an Ack. Upon retrying the request, if it reaches the majority cluster, it will receive an Ack. When the network recovers, the minority cluster will automatically become Followers [*].

If more than half of the PD nodes are damaged, we can directly refer to the scenario where all nodes are damaged, or handle it as a split-brain scenario. This is because if more than half of the nodes are damaged, the cluster cannot elect a leader. Alternatively, we can start a single node separately and then scale in or out the other nodes according to the split-brain example [*].

| username: 呢莫不爱吃鱼 | Original post link

The Raft protocol can ensure

| username: TiDBer_QYr0vohO | Original post link

Raft protocol

| username: Swan | Original post link

Learned a lot, thanks for sharing.

| username: 小于同学 | Original post link

It won’t succeed because log replication won’t receive responses from the majority of nodes.

| username: TiDBer_21wZg5fm | Original post link

Raft

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.