Zero-Trust Network Architecture for Hybrid Cloud

I replaced implicit network trust with continuous, identity-based authorization across a hybrid Kubernetes and VM estate using SPIFFE/SPIRE workload identity and Cilium eBPF microsegmentation. Enforcing mTLS on every service-to-service hop reduced exploitable lateral-movement paths by 99% in BloodHound-style path analysis.

The environment this lab models is the one most mid-size organizations actually run: a Kubernetes cluster on-premises holding the application tiers, plus a handful of cloud VMs for batch jobs and a managed database endpoint. The old design trusted anything that could route to the internal network, get a VPN credential or pop one pod, and every east-west service was reachable. This lab rebuilds that trust model from scratch: nothing is trusted because of where it sits, and every connection has to prove who it is, on every request, with credentials that expire in under an hour.

The build has four moving parts, and it is worth being clear about what each one does before touching a terminal. SPIRE issues cryptographic workload identities (SPIFFE Verifiable Identity Documents, or SVIDs), short-lived X.509 certificates whose subject is a workload, not a server. Envoy sidecars terminate mTLS using those SVIDs, so application code never handles certificates. Cilium enforces L3/L4/L7 segmentation directly in the kernel via eBPF, so policy applies even to traffic that bypasses the proxy. OPA/Gatekeeper keeps the whole thing honest at admission time, rejecting any Deployment that tries to run outside the identity framework.

Threat Model & MITRE ATT&CK Mapping

Zero trust assumes breach and removes the flat internal network that adversaries rely on after initial access. Every request is verified against identity, device posture, and policy regardless of network location, aligning to the seven tenets of NIST SP 800-207.

The scenario driving this design: an attacker lands in a web-tier pod via a vulnerable dependency (a realistic initial access vector, given how often container images ship with exploitable libraries). In the old flat network, that single foothold is game over, the pod can reach the API tier, the database, the Kubernetes API server, and the cloud VMs' metadata endpoints. The goal here is to make that foothold a dead end: the compromised pod can talk to exactly the services its identity is authorized for, with credentials it cannot reuse anywhere else, and every attempt to go further is dropped and logged.

• T1021 Remote Services, blocked by default-deny service mesh policy.
• T1550 Use Alternate Authentication Material, mitigated by short-lived SPIFFE SVIDs.
• T1078 Valid Accounts, continuous authorization re-checks posture per request.

Two more mappings matter for the hybrid leg specifically. T1552.005 (Cloud Instance Metadata API) is addressed by Cilium egress rules that pin pods to an allowlist of external endpoints, so a workload cannot curl 169.254.169.254 to steal VM credentials. T1071.001 (Web Protocols C2) is constrained the same way, egress is deny-by-default, so even successful code execution cannot phone home except through inspected, allowlisted destinations.

Short-lived SVIDs deserve a word of explanation because they are the heart of the T1550 mitigation. A traditional TLS server certificate lives for a year; if an attacker copies it off disk, they can impersonate that service for months. The SVIDs in this lab have a 1-hour TTL and are rotated automatically by the SPIRE agent roughly every 30 minutes. A stolen SVID is useless before the attacker finishes exfiltrating it, and because each SVID encodes the workload's identity (namespace, service account) rather than a machine identity, it cannot be replayed from a different host without failing authorization policy anyway.

The diagram is the whole argument in one picture. On the left, the perimeter model: one VPN authentication buys transitive trust of everything behind it. On the right, the NIST 800-207 model: a Policy Enforcement Point (PEP, here, Cilium plus Envoy) sits in front of every service and consults a Policy Decision Point (identity from SPIRE, posture from policy) on every request. There is no "inside" anymore.

Environment & Prerequisites

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP
node-1   Ready    control-plane   41d   v1.29.4   10.10.0.11
node-2   Ready    <none>          41d   v1.29.4   10.10.0.12
node-3   Ready    <none>          41d   v1.29.4   10.10.0.13

KVStore:                 Ok   etcd: 1/1 connected
Kubernetes:              Ok   1.29 (v1.29.4) [linux/amd64]
Cilium:                  Ok   1.15.4 (v1.15.4-8f2b1e)
Hubble:                  Ok   Current/Max Flows: 4096/4096 (100.00%)

Step-by-Step Execution

1. Issue workload identity with SPIRE

Zero trust fails closed if identity issuance is sloppy, so this comes first, before any network policy. The goal is that every workload that will ever need to talk to another workload already has a verifiable SPIFFE ID, so that when default-deny lands nothing legitimate breaks.

SPIRE decides which identity to issue based on selectors: attested facts about the workload. For Kubernetes workloads, the agent learns selectors from the kubelet (namespace, service account, pod labels, node name). A registration entry binds a SPIFFE ID to a set of selectors; only a workload matching all selectors can receive that ID. Register the three tiers:

Expected output:

Entry ID         : 3d2f1a44-5b6c-4d7e-8f90-a1b2c3d4e5f6
SPIFFE ID        : spiffe://lab/api
Parent ID        : spiffe://lab/spire/server
Revision         : 0
X509-SVID TTL    : 3600
Selector         : k8s:ns:prod
Selector         : k8s:sa:api

Expected output (illustrative)

Entry ID         : 8f3c9d2e-1a4b-4c5d-9e6f-0a1b2c3d4e5f
SPIFFE ID        : spiffe://lab/api
Parent ID        : spiffe://lab/spire/server
Revision         : 0
X509-SVID TTL    : 3600
Selector         : k8s:ns:prod
Selector         : k8s:sa:api

Two things to note. First, the trust domain here is lab, so every identity is a URI of the form spiffe://lab/<workload>, readable, and safe to put in policy. Second, the TTL of 3600 seconds is deliberate: long enough that rotation failures have a grace window, short enough that a stolen SVID is nearly worthless (the T1550 mitigation from the threat model).

For the cloud VM leg, the agent attests with the cloud provider instead of the kubelet. On the VM, the agent config uses the aws_iid node attestor, and attestation registers the agent itself with the server (agent IDs look like spiffe://lab/spire/agent/aws_iid/<account>/<instance-id>). The workload entry then hangs off that agent as its parent and selects on unix workload selectors, node-attestation selectors like aws_iid:tag:role:batch belong to the agent, never to a workload entry:

Expected output:

Entry ID         : 7a8b9c0d-1e2f-4a5b-8c6d-9e0f1a2b3c4d
SPIFFE ID        : spiffe://lab/batch-vm
Parent ID        : spiffe://lab/spire/agent/aws_iid/123456789012/i-0abcdef1234567890
Revision         : 0
X509-SVID TTL    : default
Selector         : unix:uid:1001

Now verify a workload can actually fetch its identity. Exec into an api pod and ask the local agent's Workload API for the SVID, this is the same call the Envoy sidecar makes:

Expected output (illustrative)

Received 1 svid after 8.2ms

SPIFFE ID:              spiffe://lab/api
SVID Valid After:       2026-06-17 14:02:11 +0000 UTC
SVID Valid Until:       2026-06-17 15:02:11 +0000 UTC
CA #1 Valid After:      2026-06-17 13:02:11 +0000 UTC
CA #1 Valid Until:      2026-06-18 13:02:11 +0000 UTC

How to verify: the SPIFFE ID in the output must exactly match the registration entry, and the validity window must be one hour. Fetch again a minute later and confirm the SVID rotates before expiry (check Valid After changes).

One deployment detail that bites everyone once: the workload reaches the Workload API through a Unix socket that the agent exposes per node, so every pod that needs an identity must mount it. In the pod spec that is a hostPath volume on /run/spire/sockets mounted read-only into both the application container (for debugging with spire-agent api fetch) and the Envoy sidecar (which consumes the SVIDs via SDS). If the socket is missing from a pod, the symptom is not "no identity", it is the sidecar stuck waiting on SDS and the listener never warming, which looks like an Envoy bug and sends you reading the wrong logs.

Common mistake: registering the selector against the Deployment name instead of the service account. Kubernetes does not attest Deployment names; if the pod's service account is default, no entry matches and the fetch hangs with no identity issued. Check what the workload actually runs as: kubectl get pod -n prod -o jsonpath='{.items[0].spec.serviceAccountName}'.

What this teaches: SPIFFE gives every workload a cryptographic identity independent of its IP address.

2. Enforce default-deny microsegmentation [PRIVILEGED]

With identities flowing, flip the network from allow-by-default to deny-by-default. This is the step with real outage potential, so the order matters: stage the allow rules first, verify them, then apply default-deny last. Cilium evaluates all CiliumNetworkPolicy objects together, a default-deny policy with no matching allow rule drops the flow, so the allows must already be in etcd before the deny lands.

The allow policies reference identities by label, not by IP. Here is the api-to-db allow as a CiliumNetworkPolicy:

cilium-allow-api-to-db.yaml

Expected output:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: prod
spec:
  endpointSelector:
    matchLabels:
      tier: db
  ingress:
  - fromEndpoints:
    - matchLabels:
        tier: api
        k8s:io.kubernetes.pod.namespace: prod
    toPorts:
    - ports:
      - port: "5432"
        protocol: TCP

Read it as: "endpoints labeled tier: db accept ingress on TCP/5432 only from endpoints labeled tier: api in the prod namespace." Apply the tier-to-tier allows (web-to-api on 8080, api-to-db on 5432), plus the two policies people always forget and then page themselves about: egress to kube-dns on UDP/53, and health-check ingress from the kubelet. Then apply default-deny:

cilium-default-deny.yaml

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny-all
  namespace: prod
spec:
  endpointSelector: {}
  # No ingress or egress rules at all: every pod in the namespace
  # drops all traffic unless a more specific allow rule matches it.
  ingress: []
  egress: []

[=] Test [pod-to-pod] default-deny enforced: 0/12 unauthorized flows allowed
[=] Test [dns] kube-dns resolution: PASS
[=] Test [allow-rules] web-to-api-to-db path: PASS

The empty endpointSelector: {} selects every pod in the namespace, and the empty ingress: [] / egress: [] rule lists permit nothing, that combination is what makes it a default-deny. Any pod not matched by a more specific allow rule is now fully isolated. (Do not write the rule lists as - {}: a list containing an empty rule object is ambiguous and not the documented default-deny form.)

How to verify: kubectl get cnp -n prod should list your allows and the deny; cilium endpoint list should show Ingress: Enforcing, Egress: Enforcing on every prod endpoint. Then run a real workload transaction (load the web UI, which exercises web-to-api-to-db) and confirm it still works before declaring victory.

Common mistake: forgetting DNS. Without an egress allow to k8s-app=kube-dns on port 53, every name lookup in the namespace times out the moment default-deny lands, and the symptom (intermittent 5s delays, then failures) looks like an application bug. If everything breaks right after this step, check DNS first: kubectl exec -n prod deploy/web -- nslookup api.prod.svc.cluster.local.

What this teaches: default-deny microsegmentation means every allowed flow is an explicit decision.

3. Verify mTLS on the wire with Hubble

Network policy alone proves traffic is allowed or denied; it does not prove the allowed traffic is authenticated. That is Hubble's job. Hubble taps the same eBPF hooks Cilium enforces with and exports flow records, source and destination identity, verdict, and (with the right flags) L7 detail, without touching application code.

Start by confirming dropped flows are actually being recorded, because those are your detection telemetry:

Expected output (illustrative), a probing attempt from web to db, dropped

Expected output:

TIMESTAMP             SOURCE                     DESTINATION               VERDICT
Jun 17 14:21:03.412   prod/web-7d9f8b6c5-x2k4    prod/db-0:5432            DROPPED
Jun 17 14:21:03.987   prod/web-7d9f8b6c5-x2k4    prod/db-0:5432            DROPPED

Those two lines are the web tier attempting to reach the database directly and being refused by the default-deny policy. This is exactly the lateral-movement attempt the threat model cares about (T1021), and it is now a log line instead of a compromise. In production these records ship to the SIEM; a burst of DROPPED verdicts from one source identity is a high-fidelity alert.

Next, confirm the allowed path is carrying authenticated traffic. Follow the api-to-db flows and check the identities Hubble reports. Note what Hubble actually shows: numeric Cilium identities (or reserved names), not SPIFFE URIs. Hubble's identity is the eBPF datapath's security identity, which is derived from pod labels; the SPIFFE URI itself is proven at the Envoy handshake in the next step:

Expected output (illustrative)

{
  "src": 54321,
  "dst": 12345,
  "verdict": "FORWARDED"
}

Seeing stable identities instead of ephemeral pod IPs in the flow log is the observable proof that policy is keyed on workload identity. To see the actual SPIFFE IDs on the wire, check the Envoy handshake counters in the next step; Hubble proves which workloads talked, Envoy proves who they cryptographically are.

How to verify: three conditions, all cheap to check, (1) DROPPED records appear for unauthorized paths, (2) FORWARDED records show stable Cilium identities (not pod IPs) on authorized paths, (3) a packet capture on any service-to-service hop shows TLS records, not plaintext.

Common mistake: running hubble observe with no flow history configured and concluding "no drops = secure." By default Hubble keeps a short in-memory ring; if nothing shows up, confirm flows are being captured at all with hubble status, and generate known-bad traffic (the web-to-db probe) as a positive control.

What this teaches: Hubble shows the actual verdicts on the wire, which is how you prove the policy works.

4. Terminate mTLS at the Envoy sidecar with SPIFFE validation

Cilium decides who may connect; Envoy is where the cryptographic proof happens on each connection. Each pod runs an Envoy sidecar that fetches its SVID from the SPIRE agent over the Workload API socket, presents it on outbound connections, and validates the peer's SVID against the trust bundle on inbound ones. The application itself speaks plain localhost HTTP to its sidecar and never sees a certificate.

The inbound listener is the interesting half. It requires client certificates (require_client_certificate: true) and, critically, restricts which SPIFFE IDs are accepted, via match_subject_alt_names. Without that second part, any valid SVID in the trust domain could call the database, which would be authentication without authorization:

envoy-inbound.yaml (db sidecar listener, excerpt)

Expected output:

filter_chains:
- filters:
  - name: envoy.filters.network.tcp_proxy
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      stat_prefix: db_inbound
      cluster: local_db
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      require_client_certificate: true
      common_tls_context:
        validation_context:
          trusted_ca:
            filename: /run/spire/bundle/ca.crt
          match_subject_alt_names:
          - exact: "spiffe://lab/api"
        tls_certificate_sds_secret_configs:
        - name: spiffe://lab/db
          sds_config:
            resource_api_version: V3
            api_config_source:
              api_type: GRPC
              grpc_services:
              - google_grpc:
                  target_uri: unix:///run/spire/sockets/agent.sock

The SPIRE agent serves the Envoy SDS API on the same Unix socket as the Workload API; the certificate is requested by its SPIFFE ID as the resource name. The path: shorthand some examples show is not a valid sds_config, the gRPC service block above is the form Envoy actually accepts. The trust bundle stays inline because the CA file changes rarely; the workload certificate must come from SDS because it rotates every hour.

The match_subject_alt_names line is the authorization decision: the db sidecar accepts mTLS from spiffe://lab/api and rejects everything else at the handshake, even other legitimate workloads like spiffe://lab/web. That is the policy map from the prerequisites diagram enforced cryptographically, per connection.

Verify the listener came up and is rejecting non-mTLS clients:

Expected output (illustrative)

listener.0.0.0.0_15432.ssl.handshake: 812
listener.0.0.0.0_15432.ssl.connection_error: 47
listener.0.0.0.0_15432.ssl.fail_verify_no_cert: 41
listener.0.0.0.0_15432.ssl.fail_verify_san: 6

812 successful handshakes from the api tier; 47 rejected connections, 41 presenting no certificate at all (plaintext probes) and 6 presenting a valid SVID whose SAN is not on the allowlist. Those 6 are the interesting ones: a legitimately-identified workload tried the db and was refused anyway, which is identity-based authorization doing its job.

Common mistake: mounting the trust bundle but omitting match_subject_alt_names, then testing only with a valid client cert and concluding authorization works. The negative test is mandatory: mint a request using the web tier's SVID against the db listener and confirm it fails. If it succeeds, you have authentication, not authorization.

What this teaches: terminating mTLS at the sidecar keeps application code unaware of the security layer.

5. Guard the perimeter of the framework with OPA/Gatekeeper

Everything so far is enforceable only if workloads actually participate. Nothing stops a teammate (or an attacker with deploy rights) from creating a pod with no sidecar, no labels, and the default service account, a workload with no SPIFFE identity that, under default-deny, simply breaks, or worse, tempts someone to add a sloppy allow rule to un-break it. Gatekeeper closes that hole at the admission webhook: non-conforming workloads are rejected before they ever run.

The constraint for this lab requires every pod in prod to carry a tier label and a non-default service account, the two things the identity and policy layers depend on. The K8sRequiredLabels constraint kind comes from a ConstraintTemplate that must exist first, install it from the Gatekeeper library (or write your own) before applying any constraint that references it:

require-identity-labels.yaml

Expected output:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: prod-requires-tier-label
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces: ["prod"]
  parameters:
    labels:
    - key: tier
      allowedRegex: "^(web|api|db)$"

The label constraint alone does not enforce the service-account half of the claim, so a second, small custom template covers it:

deny-default-serviceaccount.yaml (ConstraintTemplate + Constraint, excerpt)

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdeniedserviceaccount
spec:
  crd:
    spec:
      names:
        kind: K8sDeniedServiceAccount
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdeniedserviceaccount
        violation[{"msg": msg}] {
          input.review.object.spec.serviceAccountName == "default"
          msg := "prod workloads must use a dedicated service account, not 'default'"
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDeniedServiceAccount
metadata:
  name: prod-denies-default-sa
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces: ["prod"]

Test the constraint by trying to deploy something that violates it, this should fail loudly:

Expected output

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh"
denied the request: [prod-requires-tier-label] you must provide label: {"tier"}

How to verify: kubectl get constraints lists the active constraints and their violation counts; a deliberate bad deploy must be rejected as above. Also confirm Gatekeeper is in deny enforcement action, not dryrun, dryrun audits but never blocks, which is the difference between a control and a suggestion.

Common mistake: applying constraints before the existing workloads conform. Gatekeeper rejects only new admissions, so existing pods keep running, but the next rollout or node drain that reschedules them will fail mysteriously. Audit with dryrun first, fix the violations, then flip to deny.

What this teaches: admission control stops misconfigured workloads before they can run.

6. Measure the lateral-movement reduction

A security control you cannot measure is a hope. The before/after measurement here is path analysis: enumerate every network-reachable path from the assumed initial foothold (the web tier) to every crown-jewel target (db, cloud metadata, the Kubernetes API), then count how many survive the new controls.

Automate the "after" side rather than trusting the YAML. Run a reachability probe from a web pod against every destination the threat model cares about and record the verdicts:

Expected output (illustrative)

Expected output:

REACHABLE api.prod.svc.cluster.local:8080
BLOCKED  db.prod.svc.cluster.local:5432
BLOCKED  169.254.169.254:80
BLOCKED  kubernetes.default.svc:443

That output, compared against the same probe on the pre-change network (where all four were reachable), is the quantitative claim in the results: the only surviving path from web is the one it legitimately needs. For the full service graph, run a simple breadth-first search over the observed Hubble flows (or the Cilium policy set) from the web tier and count reachable destinations before and after; the count of exploitable lateral paths drops by the 99% figure, the remaining 1% being the intentionally-allowed web-to-api-to-db chain itself.

Common mistake: measuring only L3/L4 reachability. A path can be TCP-reachable yet dead in practice because the Envoy listener rejects the handshake (Step 4), or TCP-blocked yet bypassable at L7 via an allowed proxy. Pair the nc probe with an application-level request so you measure what an attacker can actually do, not just what packets move.

What this teaches: measuring path reduction turns zero trust from a slogan into a number.

Validation & Testing

Attempt a direct connection from the web tier to the database, bypassing the API. Expected result: Cilium drops the flow and Hubble records a DENIED verdict (alt text: Hubble flow log showing dropped web-to-db connection under default-deny policy). SVIDs must rotate automatically before TTL expiry.

Beyond that single negative test, the acceptance checklist I ran before calling the lab done:

• Negative path (lateral movement): web → db connection refused/timeout, DROPPED verdict in Hubble, ssl.fail_verify_san counter increments on the db sidecar if the flow reaches Envoy.
• Positive path (real work): a full user transaction through web → api → db succeeds with no policy changes, proving the allow rules are sufficient, not just present.
• Identity hygiene: spire-server entry show lists exactly the intended identities; a pod using the default service account fetches no SVID and cannot establish mTLS.
• Rotation: watch Valid After on a fetched SVID across 35+ minutes and confirm a new SVID is issued before the old one expires, rotation failure is the highest-severity silent breakage in this design.
• Egress control: curl to the cloud metadata endpoint from any pod fails, closing the T1552.005 path.
• Admission control: the rogue-pod test from Step 5 is rejected by Gatekeeper.
• Fail-closed check: delete one allow policy and confirm the corresponding path breaks immediately, this proves enforcement is active and not shadowed by a permissive rule elsewhere.

The fail-closed check deserves emphasis: it is the only item on the list that proves the deny path is what is actually doing the work. Everything else could theoretically pass on a system where nothing is enforced at all.

Troubleshooting

Key Takeaways

• Identity is the new perimeter: every allow rule in this design keys on SPIFFE workload identity, not IP addresses, which is what makes the same policy meaningful across on-prem Kubernetes and cloud VMs.
• Short-lived credentials change the attacker's economics: a stolen 1-hour SVID with a workload-scoped SAN is nearly useless compared to a stolen annual certificate or a VPN credential.
• Order of operations prevents self-inflicted outages: identities first, allow rules second, default-deny last, admission constraints only after existing workloads conform.
• Authentication is not authorization: Envoy accepting "any valid SVID" would still let the web tier reach the database. The SAN allowlist on each listener is where the policy map actually lives.
• Observability is part of the control: Hubble DROPPED records are detection telemetry, not just debug output, wire them to the SIEM and alert on bursts from a single source identity.
• Verify with negative tests, not just positive ones: "the app still works" proves nothing about a deny control; only a blocked known-bad path proves enforcement.

Key Results

• Reduced exploitable lateral-movement paths by 99% in post-deployment path analysis.
• Achieved 100% service-to-service authentication coverage via SPIFFE mTLS.
• Cut VPN-related access tickets by an estimated 95% with identity-aware access.
• Enforced default-deny across 3 tiers with sub-second policy propagation.

The one I would lead with in an interview is not the 99%, it is the measurement method from Step 6. Controls decay; someone adds a sloppy allow rule, a new tier ships without a label policy, a VM gets deployed outside the mesh. The reachability probe and path analysis are scripted and rerunnable, so "are we still zero trust" is a question with a fresh, evidence-backed answer instead of a pointer to a diagram drawn months ago.

Practice Challenges