Leader election in Kubernetes uses a Lease object in the coordination.k8s.io API so that only one replica of a controller is active at a time; standby replicas watch the Lease and take over if the leader stops renewing it. Skip it and you get split-brain: two replicas both convinced they are in charge, both acting on the same resources, each individually correct and collectively dangerous.
Every leader election tutorial ends in the same place: a Lease object exists, a client-go snippet compiles, and the post stops. It never says what happens when a real controller runs the pattern in production, under a rolling update, during a node drain, mid-failover. We run one. This post is one piece of a series on running production-grade Kubernetes preview environments, and it stays narrowly on the controller doing the routing: Gatekeeper, Autonoma's open-source (MIT) scale-to-zero engine (not to be confused with open-policy-agent/gatekeeper, the unrelated CNCF policy controller). Getting Gatekeeper from "the Lease exists" to "this is safe to run against every preview namespace in the cluster" meant making four decisions the tutorials never mention.
Why a controller needs leader election
Split-brain is the failure mode leader election exists to prevent: two replicas of the same controller, both believing they are the one in charge, both reconciling the same resources at the same time. Neither replica is wrong from where it sits. Each sees a resource that needs an action and takes it. The damage comes from both taking it: two replicas scaling the same Deployment in opposite directions, two replicas writing conflicting status fields on the same object, two reconcile loops racing to be the last writer. A stateless web server can run N identical replicas behind a load balancer with no coordination problem, because any replica can answer any request. A controller that reconciles cluster state cannot. It needs exactly one active writer at a time, everyone else standing by, ready but not acting. That pattern is active-passive, and Kubernetes gives you a primitive for it instead of making every team invent one.
The mechanism: the Lease API and client-go
The primitive is the Lease object, part of the coordination.k8s.io/v1 API group. A Lease has a small, specific spec: holderIdentity records which replica currently holds it, usually the pod name plus a UUID; leaseDurationSeconds sets how long a candidate must see no update past the last renewTime before it is allowed to force-acquire; renewTime is the timestamp the current leader updates on every successful renewal; acquireTime and leaseTransitions track when the lease was last taken and how many times it has changed hands, mostly useful for observability rather than the election logic itself.
Nobody hand-rolls the renewal loop. client-go's leaderelection package (k8s.io/client-go/tools/leaderelection) wraps a specific Lease with resourcelock.LeaseLock and drives it against three timers: LeaseDuration (how long since the last observed renewal before a standby may attempt to take over, 15 seconds by default in the library), RenewDeadline (how long the current leader keeps retrying a renewal before giving up on itself, 10 seconds by default), and RetryPeriod (how often standbys poll while waiting, 2 seconds by default). Those are current client-go defaults, not a Kubernetes API guarantee, and the package itself is still marked alpha in its own documentation, so pin behavior to the version you actually vendor rather than assuming it forever. Your code hooks in through two callbacks: OnStartedLeading fires once you have won, OnStoppedLeading fires the moment you lose the lease for any reason, including a renewal that simply failed to land in time. Every leader-shaped codepath has to treat OnStoppedLeading as "stop now," because by the time it fires another replica may already be renewing.
RBAC is easy to get wrong quietly rather than loudly: the ServiceAccount running the controller needs, at minimum, get, create, and update on leases in coordination.k8s.io. Broader controller-runtime managers commonly grant list, watch, patch, and delete as well, but those three verbs are what the leaderelection package strictly exercises. Miss one and the failure is not a crash, it is a controller that silently never becomes leader and never tells you why.
That mechanism is table stakes. It gets you a Lease, a leader, and a renewal loop. It does not tell you how to wire that leadership into a Service that routes real traffic, a Deployment that still needs to roll out, a disruption budget that either does nothing or actively gets in your way, or a recovery path for the moment the leader actually dies.
The production design: four choices that surprised people
We ran Gatekeeper one instance per preview namespace at first. That was fine at low namespace counts and progressively less fine as it grew, so we consolidated to a single cluster-mode install: three replicas, leader-elected through the mechanism above, managing every preview namespace in the cluster from one active leader. Getting that right past a demo surfaced four choices nobody's tutorial walks you through.
The Service routes on a pod label only the leader carries; standbys watch the same Lease and fail closed until one of them acquires it.
Route on a label, not on readiness
Gatekeeper's Service selects on a pod label, gatekeeper.dev/role=leader, that only the current leader carries. It deliberately does not gate the Service's endpoints on the pod being both ready and the leader. The obvious-looking alternative, requiring "ready AND leader" for a pod to receive traffic, works fine until a rolling update touches the Deployment. A rollout can only proceed by cycling pods that are legitimately not ready yet; if two of three replicas are healthy but simply not currently leading, and the routing logic treats "not leading" the same as "not ready," those two replicas look permanently unready to the Deployment controller. A rolling update against that Deployment can never complete, because it is waiting on a readiness signal that two-thirds of the fleet will never legitimately produce. Decoupling readiness from leadership keeps ordinary Kubernetes rollout mechanics working the way they are supposed to.
Non-leaders fail closed
Any replica that is not the currently seeded leader answers proxied requests with an explicit HTTP 503 and a Retry-After header. It does not accept the request and do nothing, and it does not forward it somewhere ambiguous. Failing closed means the caller gets a typed, honest "not me, try again shortly" instead of a response that looks successful but reflects no real backend logic, which is a much harder failure to debug three hops downstream.
No PodDisruptionBudget, on purpose
With exactly one traffic-carrying pod at any moment, a PodDisruptionBudget can only do one of two things: nothing, if it tolerates losing that one pod, or actively block node drains and cluster maintenance, if it does not. Neither buys real availability. Failover, not pod survival, is what makes this system tolerant of disruption: when the leader pod disappears for any reason, a standby already watching the same Lease acquires it and becomes the new leader. A PDB only gets in the way of that mechanism during legitimate cluster operations, for a guarantee the Lease already provides more cheaply.
A new leader re-derives state, it doesn't inherit it
Gatekeeper's leader owns more than routing. It tracks which preview namespaces are asleep, which are due to wake, and the idle timers driving those decisions, all in memory. When a new leader takes over, it does not attempt to reconstruct the outgoing leader's in-memory state through any handoff, because there may be nothing to hand off if the old leader died without warning. Instead it queries the cluster directly (namespace state, deployment replica counts, last-request timestamps) and rebuilds sleep and wake state from what the cluster actually shows, before it starts serving anything. That rebuild is what makes a leader crash tolerable rather than merely survivable: correctness never depends on one replica's memory of what it was doing.
| Choice | Naive tutorial | Production design |
|---|---|---|
| Traffic routing | Gate readiness on leadership | Route via label, readiness stays independent |
| Non-leader requests | Left undefined | Fail closed: 503 + Retry-After |
| Disruption handling | Add a PodDisruptionBudget | No PDB; failover is the tolerance |
| Post-failover state | Assume it carries over | Re-derive state from cluster first |
We publish Gatekeeper standalone under MIT so any team running its own Kubernetes previews can run this same active-passive design without rebuilding it from a client-go snippet. It also runs, unmodified, inside PreviewKit, Autonoma's managed preview-environment product, as part of the production preview-cluster infrastructure we operate for customers. Gatekeeper's MIT license covers the engine; it says nothing about the platform around it, which is Business Source License 1.1 today and converts to Apache 2.0 in 2028.
Running a leader-elected controller correctly is one of many things that stand between you and reliable preview environments. If you would rather not operate all of it, that is what our managed preview environments are for. Grab 20 min with a founder
Failover, step by step
Failover is Lease timeout plus startup: the new leader re-derives sleep and wake state from the cluster before it serves a single request.
When the leader dies (a crash, an OOM kill, a node failure, a plain kubectl delete pod), both standbys are already watching the same Lease. Once renewTime plus leaseDurationSeconds passes with no update, roughly 15 seconds under Gatekeeper's defaults, each standby attempts to acquire it. client-go's conditional update to the Lease object prevents both from believing they won at once; whichever acquire call lands first becomes the new leader. The gatekeeper.dev/role=leader label moves to that replica, the Service's endpoints follow the label, and the new leader immediately re-derives sleep and wake state from the cluster before it answers a single request, absorbing whatever load the standbys had been fail-closing with 503s in the meantime.
There is a related cutover case worth naming, because it fails the same way if you get it wrong. Moving a preview from its own in-namespace controller onto Gatekeeper's central cluster-mode instance is not a simple ingress repoint. Our nginx setup routes an exact-host Ingress ahead of a wildcard, so the move happens by deleting the preview's exact-host Ingress, letting traffic fall through to the central wildcard Ingress, in the same atomic step as tearing down the old in-namespace controller. Do those two operations separately and there is a window where both controllers believe they own the namespace, which is the same split-brain failure mode leader election exists to prevent, just relocated to the boundary between two controllers instead of living inside one.
None of this is specific to Gatekeeper. Any controller running active-passive against shared cluster state, an ingress controller doing custom routing, a custom scheduler, anything reconciling per-namespace resources across a cluster that also needs its namespaces properly isolated from each other, runs into the same four questions the moment it moves from a demo to a cluster-mode install serving real traffic. Consolidating Gatekeeper to cluster-mode is also what let us cut its footprint; the scale-to-zero numbers from that consolidation, a 67% reduction worth $218K a year on our own preview infrastructure, come from the same leader-elected controller described here, just measured from the cost side instead of the reliability side.
Operating a correctly designed leader-elected controller well is real, ongoing engineering work, and it is only one piece of what running your own Kubernetes preview environments actually costs alongside scale-to-zero, egress policy, and namespace sprawl. If you would rather build and own that yourselves, Gatekeeper is there to fork. If you would rather not run all of it yourselves, that is exactly what PreviewKit, Autonoma's managed preview-environment product, is for.
FAQ
The Lease API is a resource in the coordination.k8s.io/v1 group used for time-bound coordination between replicas. Its spec tracks holderIdentity (who holds it), leaseDurationSeconds (how long others must wait past the last renewal before taking over), renewTime (the last renewal timestamp), and acquireTime and leaseTransitions for history. Kubernetes uses Leases internally for node heartbeats and control-plane leader election, and any application controller can use the same object for its own leader election.
Multiple replicas of a controller race to hold a shared Lease object. Whichever replica successfully creates or updates the Lease with its own holderIdentity becomes the leader and periodically renews renewTime to keep the lease. Standby replicas watch the same Lease; if renewTime stops advancing for longer than leaseDurationSeconds, a standby attempts to acquire it. client-go's leaderelection package (resourcelock.LeaseLock) handles the acquire, renew, and release logic, and exposes OnStartedLeading and OnStoppedLeading callbacks so your code can react to leadership changes.
Not if leadership is also a routing condition. Gating a Service's endpoints on both readiness and leadership means every non-leader replica looks permanently unready, which blocks Deployment rollouts from ever completing, since a rolling update depends on cycling pods through a genuine not-ready-then-ready transition. The safer design routes traffic via a pod label that only the current leader carries, leaving the standard readiness probe independent of leadership status.
Generally no. With exactly one pod carrying traffic at a time, a PodDisruptionBudget can only tolerate losing that pod (in which case it does nothing useful) or block it from being evicted (in which case it interferes with legitimate node drains and cluster maintenance). Leader election already provides the real availability guarantee: when the active pod goes away, a standby acquires the Lease and takes over. A PDB adds friction without adding protection in this specific topology.
With client-go's default LeaseDuration of 15 seconds, a standby can acquire an abandoned Lease roughly 15 seconds after the previous leader stops renewing it. The new leader then needs to finish whatever startup work it requires, such as re-deriving state from the cluster, before it should start serving traffic. Total time to a fully-serving new leader is therefore the Lease timeout plus that startup cost, not the Lease timeout alone.
