You will expose an app through an Ingress and lock down pod-to-pod traffic with a default-deny NetworkPolicy. By the end external users reach your service while only authorized pods can talk to the backend.

Kubernetes networking is flat by design: every pod can reach every other pod on every node, with no NAT and no built-in firewall. That is great for developer velocity and terrible for security. This lab walks the full lifecycle of getting traffic into the cluster safely (Services and Ingress) and then constraining how traffic moves inside it (NetworkPolicy). You will work with a three-tier demo app, a public web frontend, an internal api, and a db backend, so every rule you write maps to a real allow-list decision.

Learning Objectives

  • Compare ClusterIP, NodePort, and Ingress for exposure.
  • Route external traffic via an Ingress controller.
  • Enforce default-deny with NetworkPolicy.
  • Verify service discovery through CoreDNS from inside a pod.
  • Prove segmentation works by testing both allowed and denied paths.
  • Time: ~4 hours · Difficulty: Advanced · Prereqs: a cluster with an Ingress controller and a CNI that supports policy.

Architecture Overview

Traffic enters through the Ingress controller, which terminates the HTTP request and routes it by host and path to a Service. The Service load-balances across the api pods. Those pods are the only workloads allowed to reach the db pods, because a default-deny NetworkPolicy blocks everything else, including pods in the same namespace that have no business touching the database.

Three layers do the work here. kube-proxy programs iptables or IPVS rules so a stable Service virtual IP forwards to healthy pod endpoints. The Ingress controller (a reverse proxy such as NGINX running as a pod) watches Ingress resources and rewrites its own config to match. The CNI plugin (Calico or Cilium) enforces NetworkPolicy at the pod's virtual interface, dropping packets that match no allow rule. NetworkPolicy itself is only a declaration, without a policy-capable CNI it is silently ignored.

Environment Setup

Prerequisites

  • A local cluster (kind) with kubectl, plus an ingress controller installed.
  • The three-tier demo manifests from the lab, or any frontend, backend, and database set.
  • Assumed skills: Kubernetes basics (pods, deployments, services) and DNS fundamentals.

You will need: a cluster, an Ingress controller (ingress-nginx), and a policy-capable CNI (Calico/Cilium).

Before you begin: the default cluster allows all pod traffic until you add a policy.

First confirm your context points at the right cluster and that every node is Ready. A NotReady node will silently swallow pod scheduling later and waste your debugging time.

Check cluster access and node health

kubectl cluster-info
kubectl get nodes -o wide

Expected output

Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP
lab-cp     Ready    control-plane   12d   v1.29.4   192.168.49.2

Next, verify the CNI actually enforces policy. Look for Calico or Cilium pods in kube-system. If you are on kind or Docker Desktop with the default CNI (kindnet), NetworkPolicy will apply without error but do nothing, the classic silent failure of this lab.

Confirm a policy-capable CNI is running

kubectl -n kube-system get pods | grep -Ei 'calico|cilium'

Expected output

calico-kube-controllers-7ddc4f6b9f-x2kpv   1/1     Running   0          12d
calico-node-8m4qw                         1/1     Running   0          12d
How to verify: no output from that grep means no policy enforcement. On a fresh cluster, install Calico with kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml before continuing.
Do not install Calico over kind's default CNI. kind ships with kindnet, and applying the Calico manifest on top leaves two CNIs fighting over pod networking. Create the kind cluster with the default CNI disabled first:
# kind-calico.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: 192.168.0.0/16
kind create cluster --name lab --config kind-calico.yaml
Then install Calico on the empty CNI slot. The podSubnet must match the IP pool in the Calico manifest (192.168.0.0/16 by default); a mismatch leaves nodes without usable pod routes. Skipping the CNI swap is the most common reason policies silently do nothing on kind.

Install the ingress-nginx controller if it is not present. The manifest creates the ingress-nginx namespace, the controller Deployment, and a Service of type LoadBalancer that fronts it.

Retirement notice: the ingress-nginx project was archived by its maintainers on 2026-03-24 and is now read-only; controller-v1.15.1 is the final release and will receive no security patches. It still works for this lab, but plan new deployments on the Gateway API (for example Envoy Gateway) instead of new ingress-nginx installs.

Install ingress-nginx and wait for the controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/cloud/deploy.yaml
kubectl -n ingress-nginx wait --for=condition=ready pod -l app.kubernetes.io/component=controller --timeout=120s

On minikube use minikube addons enable ingress instead. On bare metal or kind with no cloud load balancer, the controller Service will sit at <pending> for EXTERNAL-IP, see the Troubleshooting section for the port-forward workaround.

Finally, sanity-check CoreDNS, because every Service lookup in this lab depends on it:

Verify CoreDNS is healthy

kubectl -n kube-system get pods -l k8s-app=kube-dns

Expected output

coredns-76f75df574-4znpt   1/1   Running   0   12d
coredns-76f75df574-w8qh2   1/1   Running   0   12d

Step-by-Step Execution

01
Deploy the three-tier demo app

Work in a dedicated namespace so policies are easy to reason about and cleanup is one command. Deploy web (NGINX), api (a simple HTTP responder), and db (PostgreSQL), each labelled so Services and NetworkPolicies can select them.

Create the namespace and workloads

kubectl create namespace netlab
kubectl -n netlab create deployment web --image=nginx:1.25 --port=80
kubectl -n netlab create deployment api --image=hashicorp/http-echo:0.2.3 -- -text="api-ok"
kubectl -n netlab create deployment db --image=postgres:16 --port=5432
kubectl -n netlab set env deployment/db POSTGRES_PASSWORD=labpass

Verify all pods are running

kubectl -n netlab get pods -o wide --show-labels

Expected output

NAME                   READY   STATUS    RESTARTS   AGE   IP            LABELS
api-5f8c9d7b6d-k2j9x   1/1     Running   0          40s   10.244.0.12   app=api,...
db-6c8b5f7c9-4mztq     1/1     Running   0          40s   10.244.0.13   app=db,...
web-7d9c4b8f5-rp2lw    1/1     Running   0          40s   10.244.0.11   app=web,...
Common mistake: forgetting the label. Every kubectl create deployment automatically sets app=<name>, and steps below rely on exactly those labels. If you build your own YAML later, keep app: web, app: api, and app: db consistent or your selectors will match nothing.

What this teaches: The three-tier app gives every network primitive something to connect; deploy the workloads before the wiring.

02
Expose a service internally

Pod IPs are ephemeral, they change on every restart, reschedule, or scale event. A Service gives a stable virtual IP and DNS name that load-balances across matching pods. The default type, ClusterIP, is reachable only from inside the cluster, which is the correct default for anything that is not public-facing.

kubectl -n netlab expose deploy web --port 80 --target-port 80 --name web

Do the same for api and db, then inspect what was created:

Expose remaining tiers and inspect the Services

kubectl -n netlab expose deployment api --port 8080 --target-port 5678
kubectl -n netlab expose deployment db --port 5432 --target-port 5432
kubectl -n netlab get svc

Expected output

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
api    ClusterIP   10.96.214.87    <none>        8080/TCP   10s
db     ClusterIP   10.96.108.201   <none>        5432/TCP   8s
web    ClusterIP   10.96.45.173    <none>        80/TCP     2m

Prove internal reachability by curling the Service name (not the pod IP) from a throwaway pod. CoreDNS resolves web because the test pod lives in the same namespace:

Test Service DNS from inside the cluster

kubectl -n netlab run test --rm -it --image=busybox:1.36 -- wget -qO- http://web

Expected output (NGINX welcome page, truncated)

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
How to verify: kubectl -n netlab describe svc web shows an Endpoints: list. If Endpoints is <none>, the Service selector does not match any pod labels, that mismatch, not DNS, is your problem.

What this teaches: ClusterIP is the internal default; it proves pods can find each other without any external exposure.

03
Compare ClusterIP, NodePort, and Ingress

Before going further, be deliberate about the exposure model, because it decides your attack surface:

  • ClusterIP, internal only. Use for api and db; they should never be reachable from outside.
  • NodePort, opens a high port (30000-32767) on every node. Fine for quick demos, poor for production: no TLS, no host routing, and it bypasses your edge controls.
  • Ingress, one external entry point that routes many hostnames and paths to many Services, terminates TLS, and centralizes logging and rate limiting. This is what you will use for web.

You can see the NodePort mechanics without keeping it: patch the Service, observe the allocated port, then revert so the lab continues with only Ingress as the external path.

Peek at NodePort behaviour, then revert

kubectl -n netlab patch svc web -p '{"spec":{"type":"NodePort"}}'
kubectl -n netlab get svc web   # note the 80:31xxx/TCP mapping
kubectl -n netlab patch svc web -p '{"spec":{"type":"ClusterIP"}}'
Why this matters for security: every NodePort is an unauthenticated listener on every node IP. Auditing for stray NodePorts (kubectl get svc -A | grep NodePort) is a real hygiene check on any cluster you inherit.

What this teaches: Comparing the three types side by side makes the exposure ladder concrete: internal, node port, then L7 routing.

04
Route external traffic with Ingress

An Ingress resource is just a rule set; the controller you installed in setup turns it into live reverse-proxy config. This rule sends any request whose Host header is web.lab.tyfsadik.org to the web Service on port 80.

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: web }
spec:
  ingressClassName: nginx
  rules:
    - host: web.lab.tyfsadik.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: web, port: { number: 80 } } }

Apply and confirm the rule is programmed

kubectl -n netlab apply -f ingress.yaml
kubectl -n netlab get ingress web
kubectl -n ingress-nginx get svc ingress-nginx-controller   # grab the EXTERNAL-IP

Expected output

NAME   CLASS   HOSTS                  ADDRESS        PORTS   AGE
web    nginx   web.lab.tyfsadik.org   203.0.113.50   80      15s

Since the lab hostname is not in public DNS, send the Host header explicitly against the controller's external address (203.0.113.50 below is a documentation address, substitute your controller's actual EXTERNAL-IP):

Curl through the Ingress

curl -s -H 'Host: web.lab.tyfsadik.org' http://203.0.113.50/ | head -5
Common mistake: a 404 here almost always means the Host header does not match the host: field, or the request hit a different controller. Confirm with kubectl -n netlab describe ingress web, the Events section shows whether the controller synced the rule.

What this teaches: Ingress routes by host and path; one entry point can serve many backends, which is the production pattern.

05
Verify CoreDNS service discovery

Everything above worked because CoreDNS answers names like web, web.netlab, and the fully qualified web.netlab.svc.cluster.local. Take thirty seconds to see the record directly, you will need this reflex later when debugging egress policies, because a default-deny egress rule also blocks DNS if you forget to allow it.

Resolve the Service from inside a pod

kubectl -n netlab run dnscheck --rm -it --image=busybox:1.36 -- nslookup web.netlab.svc.cluster.local

Expected output

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web.netlab.svc.cluster.local
Address 1: 10.96.45.173 web.netlab.svc.cluster.local
How to verify: the answering server should be the kube-dns ClusterIP (10.96.0.10 by default). If lookups hang, check kubectl -n kube-system logs deploy/coredns before suspecting your app.

What this teaches: CoreDNS is the service discovery backbone; if names resolve, the platform's wiring is healthy.

06
Apply a default-deny policy

Default-deny then explicit-allow is the zero-trust posture for pod traffic.

A NetworkPolicy with an empty podSelector selects every pod in the namespace, and an empty rule set means "allow nothing". The moment this applies, all ingress and egress in netlab is dropped, including DNS lookups from your pods, which is why step 07 adds DNS egress back immediately.

# default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: netlab
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
kubectl apply -f default-deny.yaml

Confirm the policy exists and selects all pods

kubectl -n netlab get networkpolicy
kubectl -n netlab describe networkpolicy default-deny | head -12

Expected output

NAME            POD-SELECTOR   AGE
default-deny    <none>         6s
Common mistake: a POD-SELECTOR of <none> is correct here, it means "all pods", not "no pods". An empty podSelector: {} selects everything; forgetting the field entirely is invalid and the API will reject it.

What this teaches: Default-deny flips the cluster from allow-all to deny-all; it is the moment network policy becomes real.

07
Poke only the holes you need

Now open exactly the paths the app needs: api pods may reach db on TCP 5432 (both the ingress half on db and the egress half on api, because default-deny blocks both directions), every pod may reach CoreDNS on UDP/TCP 53 so name resolution keeps working, and the ingress-nginx controller may reach web on port 80 so the external path from step 04 survives default-deny. Nothing else in the namespace can initiate anything.

# allow-api-to-db.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: netlab
spec:
  podSelector:
    matchLabels: { app: db }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: api }
      ports:
        - protocol: TCP
          port: 5432
---
# allow-dns.yaml - egress to kube-dns so lookups survive default-deny
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: netlab
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# allow-api-egress-to-db.yaml, the egress half default-deny also blocks

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-egress-to-db
  namespace: netlab
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: db }
      ports:
        - protocol: TCP
          port: 5432

# allow-ingress-to-web.yaml, lets the ingress controller reach web under default-deny

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-web
  namespace: netlab
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 80

Apply the allow rules

kubectl apply -f allow-api-to-db.yaml
kubectl apply -f allow-api-egress-to-db.yaml
kubectl apply -f allow-dns.yaml
kubectl apply -f allow-ingress-to-web.yaml
kubectl -n netlab get networkpolicy
How to verify: rules are additive, the union of all policies selecting a pod is what is allowed. There is no "deny" statement and no rule ordering. If traffic flows that you expected to be blocked, some other policy is allowing it; list them all with kubectl -n netlab get netpol -o wide.

What this teaches: Each allow rule is a deliberate hole; writing them forces you to state what the app actually needs.

08
Prove both the allow and the deny

Segmentation that you have not tested is a hope, not a control. From a pod labelled app=other (which no policy permits), the database connection must fail; from an api pod it must succeed.

Launch an unauthorized pod and test the DB path

kubectl -n netlab run other --rm -it --image=busybox:1.36 --labels app=other -- nc -zv -w3 db 5432 || echo denied
kubectl -n netlab run api-probe --rm -i --image=busybox:1.36 --labels app=api -- nc -zv -w3 db 5432
Why a throwaway pod instead of exec into the api pod? hashicorp/http-echo is built from scratch: it has no shell and no nc. The probe pod carries the app=api label, so the policies treat it exactly like an api pod.

Expected output

denied                                  # from the 'other' pod (labeled app=other, which no policy selects)
db (10.96.108.201:5432) open            # from an 'api' pod

The external path should still work end to end, because Ingress traffic enters through the controller in its own namespace and the web pod is only serving, not initiating:

Re-test the public path

curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: web.lab.tyfsadik.org' http://203.0.113.50/

Expected output

200
If nc from the probe pod hangs instead of printing open: the egress side is blocking it. The allow-api-to-db policy governs ingress to db, but default-deny also drops egress from api, which is why step 07 applies the separate allow-api-egress-to-db policy. If you skipped that file, apply it now. Symmetric allow rules are the most commonly forgotten half of a policy.

What this teaches: Proving both the allow and the deny is the complete test; half-tested policy is assumed policy.

Step Flow Map

Each box is one step in the lab; work them in order, since later steps build on earlier state.

Progress So Far

How It Works

Every pod gets its own IP from the cluster's pod CIDR, and every Service gets a virtual ClusterIP that exists only as load-balancing rules. kube-proxy programs those rules into iptables or IPVS on every node: a packet sent to the ClusterIP is rewritten to one of the backend pod IPs. The Service IP is stable while pod IPs churn, which is why everything inside the cluster addresses services, never pods.

The Service types are a progression of exposure. ClusterIP is reachable only inside the cluster; NodePort opens the same port on every node's IP; LoadBalancer asks the cloud for an external IP; Ingress is different in kind, an L7 reverse proxy that routes by host and path to ClusterIP services. Comparing them side by side is what makes the tradeoffs concrete instead of theoretical.

CoreDNS resolves service names to ClusterIPs from inside any pod, which is the service discovery mechanism the whole platform leans on. NetworkPolicies are the firewall layer: by default all traffic is allowed, and a default-deny policy flips the cluster to deny-all so you can open only the paths the application needs. Policies are enforced by the CNI plugin, so the exact behavior depends on which CNI your cluster runs.

Testing & Validation

Run the full acceptance check in one pass: external reachability through the Ingress, and denial of an unauthorized internal path. On kind the controller Service never gets an EXTERNAL-IP (no cloud load balancer), so the check port-forwards the controller to localhost first.

# forward the ingress controller to localhost (kind has no LoadBalancer IP) kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 & PF=$!; sleep 3 curl -s -o /dev/null -w 'ingress HTTP code: %{http_code}\n' -H 'Host: web.lab.tyfsadik.org' http://localhost:8080/ kill $PF # an unauthorized pod must NOT reach the database kubectl -n netlab run other --rm -i --image=busybox:1.36 --labels app=other -- nc -zv -w3 db 5432 && echo 'POLICY BROKEN: db is reachable' || echo denied

The curl must print ingress HTTP code: 200; anything else (000, 404, 503) means the Ingress rule, the host header, or the controller is misconfigured, stop and fix it before continuing. The probe pod must print denied; if it prints POLICY BROKEN the default-deny policy is not enforced on this cluster (usually the CNI, not the policy, is at fault on kind). Together the two checks confirm exposure plus segmentation.

For a final review, dump the effective policy set and the Ingress programming so you can read exactly what the cluster enforces:

Review the final state

kubectl -n netlab get svc,ingress,networkpolicy
kubectl -n netlab describe ingress web | tail -8
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | tail -5

Three green checks to look for: the Ingress has an ADDRESS, every NetworkPolicy lists the selectors you intended, and the controller access log shows your 200 for web.lab.tyfsadik.org. When you are done, kubectl delete namespace netlab tears down the entire lab, workloads, Services, and policies together.

Troubleshooting

Quick-reference for the failures you are most likely to hit in this lab.

SymptomLikely causeFix
Ingress returns 404Host or path rule does not match the requestInspect the ingress rules and test with an explicit Host header via curl.
Service has no endpointsThe selector does not match any pod labelsAn empty kubectl get endpoints plus kubectl get pods --show-labels reveals the mismatch.
DNS lookups fail from podsCoreDNS is down or the query is malformedCheck the CoreDNS pods in kube-system, then test with a fully qualified name.
Default-deny broke everythingDeny applied without the allow rules the app needsAdd ingress and egress allows for DNS (port 53) first; even DNS needs an explicit hole.
NodePort unreachable from outsideHost firewall or the port mapping blocks itVerify the node port with kubectl get svc and check the host firewall rules.
Troubleshooting
  • Ingress 404: the host/path rule must match the request Host header. Verify with kubectl -n netlab describe ingress web and confirm you are hitting the right controller address. A missing ingressClassName on clusters with multiple controllers causes the same symptom.
  • Policy has no effect: your CNI must support NetworkPolicy (Calico/Cilium), not all do. kind's default kindnet and some managed-node images ignore policies entirely, traffic keeps flowing and nothing logs why. Check kubectl -n kube-system get pods | grep -Ei 'calico|cilium'.
  • DNS fails inside pods: check CoreDNS pods are Running in kube-system. If DNS broke only after applying default-deny, your egress policy is blocking port 53, apply the allow-dns policy from step 07.
  • Ingress EXTERNAL-IP stuck at <pending>: there is no cloud load balancer to allocate one. On minikube run minikube tunnel in a second terminal; elsewhere, port-forward directly with kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 and test against http://localhost:8080.
  • Everything in the namespace broke after default-deny: that is the policy working, not failing. Rebuild connectivity one allow rule at a time, DNS egress first, then app-to-app paths, and test after each so you know which rule restored which flow.

Extension Ideas

  • Add TLS to the Ingress with cert-manager: install the CRDs, create a ClusterIssuer against Let's Encrypt staging, and annotate the Ingress with cert-manager.io/cluster-issuer so the certificate is issued and rotated automatically.
  • Write an egress policy that allows api to reach only one external API endpoint by CIDR, and observe how ipBlock rules interact with DNS resolution.
  • Move to eBPF policy in the Zero-Trust Hybrid Cloud lab.
  • Deploy this on managed Kubernetes in AKS.

Key Results

  • Exposed an app externally through an Ingress controller.
  • Enforced default-deny pod traffic with explicit allow rules.
  • Verified both external reachability and internal segmentation.
  • Applied a zero-trust posture inside the cluster.

Key Takeaways

  • Services give stable IPs and DNS names over ephemeral pods; ClusterIP is the safe default, NodePort and LoadBalancer widen the attack surface.
  • An Ingress is only a rule set, the controller does the work, and the Host header must match the rule exactly.
  • NetworkPolicy is enforced by the CNI, not by Kubernetes itself; without a policy-capable CNI it fails silently.
  • Default-deny first, then additive allow rules, and remember egress covers DNS too, or everything appears "down".
  • Always test both sides: an allowed path that works and a denied path that fails are the proof the segmentation is real.

Practice Challenges

Challenge 1: Two hosts, one ingress

Goal: Add a second ingress host routing to a different backend service.

Hint: One Ingress object can hold many rules; the host field is the router.

Challenge 2: Least-privilege policy

Goal: Write a NetworkPolicy allowing only frontend to backend on 5432, then prove the database rejects anything else.

Hint: Test from a temporary debug pod; its blocked connection is the proof.

Challenge 3: Compare kube-proxy modes

Goal: Check whether your cluster uses iptables or IPVS mode and inspect the programmed rules.

Hint: The kube-proxy pod logs name the mode at startup.