Automatic Failover, Fencing, and Split-Brain

The one thing to understand first

Streaming replication gives you standbys ready to take over, but deciding when to promote one and ensuring only one primary exists is the hard part. Doing it by hand is slow and error-prone. Tools like Patroni, repmgr, and Stolon automate failover — but automation introduces the cluster's most dangerous failure mode: split-brain.

Automated failover is not really about promoting a standby quickly — it is about guaranteeing only one primary ever exists. Every mechanism here (leader lock, self-demotion, fencing, quorum) is in service of preventing two nodes from both accepting writes.

The split-brain nightmare

Split-brain is when two nodes both believe they are the primary and both accept writes. Clients write conflicting data to each; when the network heals, there is no safe way to merge the divergent histories — you have irreconcilable data loss. Every automated failover design exists primarily to prevent split-brain, not merely to speed up promotion.

The DCS and leader election

Patroni-style tools rely on a Distributed Configuration Store (DCS) — etcd, Consul, or ZooKeeper — that provides reliable consensus. The current primary holds a leader lock in the DCS with a short TTL and must keep renewing it (a heartbeat). If the primary cannot renew the lock (it crashed, or is network-partitioned away from the DCS), the lock expires. Standbys watching the DCS then race to acquire the lock; the winner promotes. Because the DCS grants the lock to only one node, only one node may become primary.

Primary: renew leader lock every few seconds (TTL e.g. 30s)
  |- can't reach DCS -> can't renew -> lock expires
Standbys: watch the lock
  |- lock expired -> attempt to acquire -> single winner promotes

Why the old primary must demote itself

The subtle danger: a primary that is partitioned from the DCS but still reachable by some clients could keep accepting writes while a standby is promoted elsewhere — instant split-brain. The defence is that the primary monitors its own lock: if it cannot renew the leader lock, it must demote itself (stop accepting writes, become read-only or shut down) before the TTL expires. Patroni does exactly this — a primary that loses the DCS shuts down its own Postgres.

Fencing and STONITH

Self-demotion assumes the old primary is healthy enough to act. If it is hung or its clock is wrong, it might not demote in time. Fencing forcibly isolates a suspect node so it cannot serve clients:

• STONITH ("Shoot The Other Node In The Head") — power off or reset the old primary via IPMI/cloud API.
• Network fencing — revoke its access (firewall, detach its VIP, revoke storage).
• Resource fencing — remove the floating IP/VIP so clients can no longer reach the old primary even if it's up.

Fencing guarantees the old primary is truly out before the new one takes over — the strongest split-brain protection.

Quorum prevents minority promotion

A robust cluster has an odd number of DCS nodes (3 or 5) so the cluster can determine which side of a partition holds the majority. Only the majority side may hold the leader lock; a minority partition cannot promote, so it cannot create a second primary. This is why you never run a 2-node "HA" cluster — it cannot distinguish "the other node died" from "I am cut off," and either choice risks split-brain.

Layer 3 — Watch it happen on your own database

The whole design comes together as one sequence — trace it once and every component's job becomes obvious:

• Primary stops renewing the leader lock (failure or partition).
• Primary self-demotes; fencing isolates it if it cannot.
• A standby on the majority side wins the DCS lock and promotes (new timeline).
• Clients are redirected (VIP move, DNS, or pooler reconfiguration).
• The old node, once recovered and fenced-safe, rejoins as a standby via pg_rewind.

You can watch this live in a Patroni cluster: patronictl list shows the leader lock holder and each node's role/lag; kill the primary and watch the lock expire, a majority-side standby acquire it, and the timeline increment. The DCS (etcdctl get /service//leader) shows exactly one leader key at all times — that single key is the split-brain guarantee.

Layer 4 — The levers this hands you

Robust failover is a set of deliberate design choices, not a default — these are the levers:

• Use an odd-sized, well-separated DCS (3 or 5 nodes) for true quorum. Only the majority side may hold the leader lock, so a minority partition cannot promote a second primary. Never run a 2-node "HA" cluster — it cannot distinguish "the other node died" from "I am cut off."
• Never rely on self-demotion alone for critical systems — add fencing. Self-demotion assumes the old primary is healthy enough to act; a hung node or wrong clock may not demote in time. Fencing forcibly isolates it: STONITH (power off/reset via IPMI/cloud API), network fencing (firewall/revoke access), or resource fencing (remove the floating IP/VIP).
• Tune TTLs to balance fast failover against false positives from transient network blips.
• Route clients through a layer (VIP or pooler) that can be repointed atomically, so promotion and client redirection happen cleanly.

Layer 5 — What an Oracle DBA should expect vs what they get

An Oracle DBA will recognise these problems from RAC and Data Guard, but the Postgres answers are assembled from external pieces:

• Split-brain prevention ≈ RAC's voting disk / fencing. The DCS leader lock plays the role of RAC's quorum/voting mechanism; STONITH is the same concept (and the same acronym) Oracle Clusterware uses.
• Patroni + a DCS ≈ Data Guard Broker + Fast-Start Failover (FSFO) with an Observer. The DCS is the Observer/quorum; the leader lock is FSFO's arbitration that only one primary may exist.
• It is not built in. Where Oracle ships Clusterware/Grid Infrastructure and the Data Guard Broker as integrated products, Postgres has no native cluster manager — you assemble Patroni/repmgr + etcd/Consul + a VIP/pooler yourself. The engine provides primitives (promotion, timelines, pg_rewind); the orchestration is external.
• Quorum math is the same lesson as RAC: an odd number of voting members, no 2-node clusters — an Oracle DBA's instinct here transfers directly.

Key takeaway

Automated failover exists to guarantee a single primary, not merely to promote quickly: a DCS leader lock with quorum decides who is primary, the old primary self-demotes (or is fenced via STONITH) so it cannot keep accepting writes, and the recovered node rejoins with pg_rewind. Use an odd-sized DCS, never a 2-node cluster, and add fencing on top of self-demotion for any system where split-brain would be catastrophic.