Kubernetes NetworkPolicy best practices start with a default-deny posture for every namespace, then add explicit ingress and egress allow rules built from precise pod and namespace selectors rather than broad CIDR blocks. Default-deny matters because Kubernetes ships with no network isolation at all: without it, every pod can reach every other pod and the wider network by default. The subtle trap is egress: a broad allow rule with RFC1918 except ranges can silently block traffic you actually need, including calls to the Kubernetes API server itself.
Every best-practices list says the same six things about NetworkPolicy: default-deny, precise selectors, cover both ingress and egress, scope by namespace, check your CNI supports enforcement, and audit regularly. All true. None of them mention the one rule that will actually take down a controller in production: the egress except block that looks like it is narrowing an allow, and is actually shadowing it.
We found this the hard way while operating PreviewKit, Autonoma's managed preview-environment product, with Gatekeeper (our own scale-to-zero controller, not to be confused with the unrelated CNCF open-policy-agent/gatekeeper admission controller, a naming collision, not a shared codebase) in front of sleeping preview namespaces. The NetworkPolicy looked correct on paper and blocked Gatekeeper's API server calls with no error anywhere in the chain. Here's the checklist that should ship with every cluster, and then the specific way egress breaks that nobody documents.
NetworkPolicy best practices checklist
Start with default-deny. A namespace with no NetworkPolicy objects has no network isolation: every pod can talk to every other pod, and often the outside network too. A NetworkPolicy that selects all pods with an empty podSelector and denies both Ingress and Egress in its policyTypes gives you a clean slate. Everything else is an explicit hole you punch in that wall, so you can always answer "why can this pod talk to that one" by pointing at a specific policy.
Precision in your selectors is what keeps the policy meaningful as the cluster grows. A namespaceSelector scoped to a label like team: checkout is durable; an ipBlock matching your entire VPC CIDR is not, it is a default-allow with extra steps. Ingress and egress need equal attention: teams write tight ingress rules for the traffic they worry about attackers reaching, then leave egress wide open because "nothing is trying to break out." That asymmetry is backwards from a blast-radius standpoint, and it is exactly where our incident happened.
Namespace scoping is the difference between a policy that fails safe and one that fails open. A rule with no namespaceSelector restriction can accidentally allow traffic from every namespace in the cluster, including ones you don't control. And none of this works if your CNI doesn't enforce NetworkPolicy in the first place: the base Flannel CNI, for instance, only provides connectivity, so the policy YAML applies cleanly and does nothing. Verify enforcement, don't assume it.
Audit by asking a different question than "does this policy exist." Ask "if I deleted this policy, what traffic would newly succeed that shouldn't." That surfaces stale exceptions faster than a checklist does.
| Practice | Fails open (bad) | Fails closed (good) |
|---|---|---|
| Baseline posture | No policy in namespace | Default-deny, then explicit allows |
| Selectors | Broad ipBlock CIDR | Pod and namespace label selectors |
| Direction coverage | Ingress only | Ingress and egress both denied by default |
| Namespace scope | Missing namespaceSelector | Explicit namespace or label scope |
| CNI check | Assumed enforced | Verified via CNI docs or a test policy |
| Egress excepts | Broad allow with RFC1918 except | Specific allow, no wide except needed |
With no policy a namespace fails open and every pod is reachable; default-deny fails closed, so each connection is an explicit allow you punch through the wall.
Ingress vs egress: where teams get it wrong
Ingress rules get the scrutiny because ingress is where the mental model of "attacker gets in" lives. Teams write a tight NetworkPolicy for their public-facing service, lock down which namespaces can reach it, and call the job done. Egress gets treated as an afterthought, usually solved one of two ways: left completely open (no egress policy type declared, so every pod can reach the internet, the metadata service, and every other pod), or over-corrected into one sweeping rule meant to cover "everything this pod should ever need."
Teams lock down ingress and leave egress wide open, which is backwards: the neglected direction is where lateral movement and the API-server egress trap actually live.
That second failure mode is worse than the first, because it looks like good practice. A platform team writes one broad egress rule allowing 0.0.0.0/0 with a handful of except entries carving out the private RFC1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), reasoning that pods should be able to reach the public internet for package registries and third-party APIs, but not pivot into other private subnets. It reads as the responsible middle ground: broad enough to be low-maintenance, narrow enough to block lateral movement.
The problem is what that rule does to traffic the pod actually needs inside those excepted ranges, including the Kubernetes API server itself, which almost always lives on an RFC1918 address.
The egress gotcha that broke our controller
Gatekeeper has to reach the Kubernetes API server to do its job: watching for requests to sleeping preview namespaces, listing pods on wake, and scaling workloads back up. Under a strict default-deny egress posture with no allow for the API server's address, those calls simply time out, and that failure mode is obvious. You watch the logs, see connection timeouts, and add an egress rule for the API server CIDR. Normal troubleshooting.
The version that actually cost us time was subtler. The cluster's egress policy for that namespace included a broad 0.0.0.0/0 allow, intended to let workloads reach the public internet, with except entries for the RFC1918 ranges to block lateral movement into other private subnets. The Kubernetes API server's ClusterIP sat inside one of those excepted ranges. Gatekeeper's calls to the API server failed, silently, with nothing in kubelet logs, nothing in the API server's audit log (the request never arrived), and no admission or policy-denial event anywhere we looked. From the pod's point of view, the connection just never completed.
As we reverse-engineered it from behavior on the AWS VPC CNI, the mechanism is a longest-prefix-match conflict, not a logical AND of "allowed unless excepted." The CNI's policy agent compiles ipBlock allow and except entries into eBPF maps that resolve via longest-prefix match, the same mechanism used for standard IP routing tables. A /0 allow and a /12 except aren't evaluated as "allow this range, minus this smaller hole." They're evaluated as two separate prefix entries in the same trie, and the more specific prefix wins the lookup regardless of which rule it came from. Our API server's address matched the /12 except more specifically than it matched the /0 allow, so the deny won the match, full stop. We are describing this as observed behavior from testing our own cluster, not a documented guarantee from AWS, and CNI internals do change across versions, but it lines up with public issue trackers for the AWS network policy agent discussing exactly this kind of except-versus-allow trie precedence.
The fix has two forms, and which one you want depends on whether you can tolerate any lateral-movement risk from removing the except. The safer fix is to keep the broad 0.0.0.0/0-with-excepts policy off any pod that needs API server access, and give that pod (in our case, the Gatekeeper controller) its own narrow egress policy: allow to the API server's specific address on port 443, allow whatever else it needs, deny the rest. The other fix is to make the API-server allow more specific than the except that's shadowing it, an explicit ipBlock for the API server's exact address (often a /32) will always win a longest-prefix-match lookup against a /12 except, because it's the more specific prefix. Either way, the lesson is the same: a broad egress allow with RFC1918 excepts is not a safe default for any pod that needs to reach an address inside one of those excepted ranges, and the API server usually does.
CNI-specific behavior
This is not universal Kubernetes behavior, it's an artifact of how a specific CNI's policy engine compiles CIDR rules, and the three CNIs we get asked about most often handle it differently enough that "check your CNI" belongs on the checklist, not as a footnote.
The AWS VPC CNI has supported native Kubernetes NetworkPolicy enforcement since v1.14, through a separate aws-network-policy-agent daemonset that attaches eBPF programs to each pod's host-side veth interface. That agent's public issue tracker documents ongoing work on how except CIDRs inherit into the trie relative to their parent allow block, consistent with what we saw: except handling is an eBPF LPM-trie precedence problem, not a simple set-subtraction. Treat any policy combining a broad allow with excepts as CNI-version-sensitive on this platform.
Cilium's CiliumNetworkPolicy CIDR rules (toCIDRSet / fromCIDRSet) are explicit that an except entry is scoped only to the CIDRRule it appears in, and does not apply to any other CIDR prefix in any other rule, per Cilium's own documentation. Cilium also compiles CIDR policy into eBPF LPM-trie maps, but the except's blast radius is documented as contained to its own rule rather than global, a meaningfully different contract from what we observed on the AWS VPC CNI, worth confirming against the Cilium version you run.
Calico takes a different enforcement path. Depending on dataplane mode (iptables, eBPF, or the newer nftables dataplane), Calico's Felix agent typically resolves rules through ordered evaluation, first matching Allow or Deny wins, rather than merging all CIDR allow and except entries into one global longest-prefix structure. That rule-ordering model made this specific shadowing failure less likely to reproduce identically on Calico in our own testing, though we'd treat that as a difference in mechanism rather than a guarantee. None of the three vendors document a global "except always wins" behavior as intentional NetworkPolicy semantics, which is exactly why it's worth verifying on whichever CNI you run.
Egress gotchas like this are part of what it actually costs to run preview environments on Kubernetes yourself. If you would rather not chase them, PreviewKit, Autonoma's managed preview-environment product, handles the networking for you. Grab 20 min with a founder
Debugging blocked egress (and when a pod should fail fast)
The failure signature to watch for is a pod that's healthy by every liveness and readiness check, yet can't complete outbound calls inside your cluster's own address space. Start by confirming the CNI enforces policy at all: apply a deliberately restrictive test NetworkPolicy against a scratch pod and confirm traffic actually gets denied. Then check every NetworkPolicy that selects the pod in question, not just the one you think is relevant. Egress is additive across policies that select the same pod, and a broad cluster-wide baseline rule can combine badly with a narrower one a team applies later.
Where possible, pull whatever CNI-specific tooling exists to inspect the compiled rule set: calicoctl for Calico, cilium policy trace for Cilium, or the aws-network-policy-agent's eBPF map dumps for the AWS VPC CNI. The YAML tells you what should happen; the compiled map or trace tells you what will. When the two disagree, the compiled version wins, which is the whole shape of the bug we hit.
It's also worth distinguishing a NetworkPolicy problem from a pod lifecycle problem, since the symptoms can look similar on a dashboard. Blocked egress to a health-check dependency can produce a pod that never becomes ready and crash-loops, showing up in kubectl get pods as CrashLoopBackOff right alongside pods crash-looping for unrelated reasons (a bad image, a missing config value, an unhandled startup exception). A pod stuck terminating is a different problem, usually a PreStop hook blocked on egress it no longer has, unable to finish before its grace period runs out.
That distinction matters for anything that watches pod state and has to make a decision quickly. Gatekeeper's wake path lists every pod in a namespace before deciding it's ready to route traffic, and its default timeout is generous because some workloads genuinely take a while to start. But it doesn't wait the full timeout uniformly: a pod that's crash-looping or referencing an image that no longer exists (a leftover from an abandoned PR whose image tag got garbage-collected) is in a state that isn't going to resolve itself no matter how long you wait, so Gatekeeper gives up on that specific pod fast, typically in one to two seconds, while it keeps waiting normally on pods that are simply still starting. That's the same principle as the egress debugging above applied to orchestration: don't treat "still working on it" and "never going to work" as the same failure mode, because the correct response time is completely different.
How Autonoma Runs the Network Policy Layer Behind Every Preview
The pain this post documents isn't hypothetical for us: Autonoma builds and runs Gatekeeper as production infrastructure, not a demo, and the egress gotcha above is a bug we had to find, diagnose, and fix on our own preview cluster before we could trust the controller in front of every sleeping namespace. Getting NetworkPolicy right for a controller that needs guaranteed API server access, across whichever CNI a given cluster runs, is exactly the kind of ongoing operational surface area that doesn't show up in a features list until it breaks something.
It also isn't a one-time fix. New preview namespaces get created and torn down constantly, each one potentially inheriting a cluster-wide baseline egress policy, and every CNI upgrade is a chance for except semantics to shift underneath a policy that used to work. Treating NetworkPolicy correctness as a single audit rather than a standing operational responsibility is how a fix like ours (make the API-server allow more specific than the except shadowing it) quietly regresses six months later when someone tightens the baseline policy again for an unrelated reason.
That's the honest split in how we think about this problem. Gatekeeper itself stays MIT-licensed and open source (a separate matter from Autonoma's own platform, which is Business Source License 1.1) precisely so a team that wants to run this networking layer themselves, on their own infrastructure, can, and can read the exact egress rules it needs rather than trusting a vendor's word for it. PreviewKit, Autonoma's managed preview-environment product, is the other pole: the namespace provisioning, the wildcard DNS and TLS, and Gatekeeper's scale-to-zero proxy all run inside a managed control plane we operate, including the CNI-specific egress hardening this post covers, so a platform team doesn't have to reverse-engineer longest-prefix-match semantics on their own AWS VPC CNI before their first production incident.
Map that back to the checklist above and it's the same six items, just handled once instead of per cluster. Default-deny and precise selectors ship as the baseline for every namespace PreviewKit provisions, not something a team writes fresh for each preview. The ingress-versus-egress asymmetry this post opens with, teams securing ingress and neglecting egress, is exactly the gap the API-server allow rule closes on day one, before any team-specific policy gets layered on top. And the CNI-specific behavior in the section above this one is precisely why that allow rule gets written narrow and specific rather than broad-with-excepts: we've already paid the cost of finding where the longest-prefix-match trap sits on the CNIs we run, so a platform team adopting PreviewKit doesn't have to rediscover it the way we did. Whether the "build it yourself" pole or the "run it managed" pole is the right call for your team depends on whether owning this exact class of networking bug is a good use of your platform engineers' time, and for the hub post on Kubernetes preview environments covering the full scale-to-zero setup, including how we made the controller itself highly available with Kubernetes leader election over the Lease API, the tradeoff comes up again in more depth. The scale-to-zero case study has the cost numbers behind running Gatekeeper on our own preview infrastructure.
FAQ
Start every namespace with a default-deny NetworkPolicy that blocks both ingress and egress, then add explicit allow rules using precise pod and namespace selectors instead of broad CIDR blocks. Cover ingress and egress equally (egress is usually the neglected half), scope policies to specific namespaces rather than leaving them cluster-wide, confirm your CNI actually enforces NetworkPolicy (not every CNI does by default), and audit regularly by asking what traffic would newly succeed if a given policy were deleted.
The most common cause is an egress policy with no explicit allow for the API server's ClusterIP or endpoint address, which is straightforward to spot from connection timeouts. The subtler cause is a broad egress allow (like 0.0.0.0/0) that includes an except entry for RFC1918 private ranges, since the API server usually lives inside one of those ranges. On CNIs that compile CIDR rules into longest-prefix-match structures, that except can be evaluated as more specific than the broad allow and win the match, blocking the API server with no error surfaced anywhere in the request path.
The AWS VPC CNI has supported native Kubernetes NetworkPolicy enforcement since v1.14 through the aws-network-policy-agent daemonset, which attaches eBPF programs to each pod's host veth interface and compiles allow and except CIDR entries into eBPF maps that resolve via longest-prefix match. In our testing, an except entry for a smaller, more specific CIDR range shadowed a broader allow covering that range, which is consistent with public discussion on the agent's issue tracker about except-to-allow precedence in its trie structure. Treat this as version-sensitive behavior specific to this CNI's implementation, not documented universal NetworkPolicy semantics.
First confirm your CNI enforces NetworkPolicy at all by applying a deliberately restrictive test policy against a scratch pod. Then check every NetworkPolicy that selects the affected pod, since egress rules from multiple policies are additive and can combine unexpectedly. Use CNI-specific inspection tools where available (calicoctl for Calico, cilium policy trace for Cilium, or eBPF map dumps for the AWS VPC CNI's network policy agent) to see the compiled rule set rather than just the YAML, since the compiled version is what actually governs traffic.
Indirectly, yes. NetworkPolicy itself doesn't crash a pod, but if a pod's startup or health-check logic depends on egress that a policy is silently blocking (a database connection, a dependency's health endpoint, an API server call), the pod can fail its readiness or liveness checks and get restarted repeatedly, which surfaces as CrashLoopBackOff. It's worth ruling out egress as a cause before assuming the crash is purely an application bug, especially if the same image runs fine in a namespace without the restrictive policy applied.
