Kubernetes Networking: Services, Ingress, and NetworkPolicy
You will expose an app through an Ingress and lock down pod-to-pod traffic with a default-deny NetworkPolicy. By the end external users reach your service while only authorized pods can talk to the backend.
Kubernetes networking is flat by design: every pod can reach every other pod on every node, with no NAT and no built-in firewall. That is great for developer velocity and terrible for security. This lab walks the full lifecycle of getting traffic into the cluster safely (Services and Ingress) and then constraining how traffic moves inside it (NetworkPolicy). You will work with a three-tier demo app, a public web frontend, an internal api, and a db backend, so every rule you write maps to a real allow-list decision.
Learning Objectives
- Compare ClusterIP, NodePort, and Ingress for exposure.
- Route external traffic via an Ingress controller.
- Enforce default-deny with NetworkPolicy.
- Verify service discovery through CoreDNS from inside a pod.
- Prove segmentation works by testing both allowed and denied paths.
- Time: ~4 hours · Difficulty: Advanced · Prereqs: a cluster with an Ingress controller and a CNI that supports policy.
Architecture Overview
Traffic enters through the Ingress controller, which terminates the HTTP request and routes it by host and path to a Service. The Service load-balances across the api pods. Those pods are the only workloads allowed to reach the db pods, because a default-deny NetworkPolicy blocks everything else, including pods in the same namespace that have no business touching the database.
Three layers do the work here. kube-proxy programs iptables or IPVS rules so a stable Service virtual IP forwards to healthy pod endpoints. The Ingress controller (a reverse proxy such as NGINX running as a pod) watches Ingress resources and rewrites its own config to match. The CNI plugin (Calico or Cilium) enforces NetworkPolicy at the pod's virtual interface, dropping packets that match no allow rule. NetworkPolicy itself is only a declaration, without a policy-capable CNI it is silently ignored.
Environment Setup
Prerequisites
- A local cluster (kind) with
kubectl, plus an ingress controller installed. - The three-tier demo manifests from the lab, or any frontend, backend, and database set.
- Assumed skills: Kubernetes basics (pods, deployments, services) and DNS fundamentals.
You will need: a cluster, an Ingress controller (ingress-nginx), and a policy-capable CNI (Calico/Cilium).
Before you begin: the default cluster allows all pod traffic until you add a policy.
First confirm your context points at the right cluster and that every node is Ready. A NotReady node will silently swallow pod scheduling later and waste your debugging time.
Check cluster access and node health
kubectl cluster-info
kubectl get nodes -o wide
Expected output
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
NAME STATUS ROLES AGE VERSION INTERNAL-IP
lab-cp Ready control-plane 12d v1.29.4 192.168.49.2
Next, verify the CNI actually enforces policy. Look for Calico or Cilium pods in kube-system. If you are on kind or Docker Desktop with the default CNI (kindnet), NetworkPolicy will apply without error but do nothing, the classic silent failure of this lab.
Confirm a policy-capable CNI is running
kubectl -n kube-system get pods | grep -Ei 'calico|cilium'
Expected output
calico-kube-controllers-7ddc4f6b9f-x2kpv 1/1 Running 0 12d
calico-node-8m4qw 1/1 Running 0 12d
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml before continuing.# kind-calico.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true
podSubnet: 192.168.0.0/16
kind create cluster --name lab --config kind-calico.yaml
Then install Calico on the empty CNI slot. The podSubnet must match the IP pool in the Calico manifest (192.168.0.0/16 by default); a mismatch leaves nodes without usable pod routes. Skipping the CNI swap is the most common reason policies silently do nothing on kind.Install the ingress-nginx controller if it is not present. The manifest creates the ingress-nginx namespace, the controller Deployment, and a Service of type LoadBalancer that fronts it.
controller-v1.15.1 is the final release and will receive no security patches. It still works for this lab, but plan new deployments on the Gateway API (for example Envoy Gateway) instead of new ingress-nginx installs.Install ingress-nginx and wait for the controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/cloud/deploy.yaml
kubectl -n ingress-nginx wait --for=condition=ready pod -l app.kubernetes.io/component=controller --timeout=120s
On minikube use minikube addons enable ingress instead. On bare metal or kind with no cloud load balancer, the controller Service will sit at <pending> for EXTERNAL-IP, see the Troubleshooting section for the port-forward workaround.
Finally, sanity-check CoreDNS, because every Service lookup in this lab depends on it:
Verify CoreDNS is healthy
kubectl -n kube-system get pods -l k8s-app=kube-dns
Expected output
coredns-76f75df574-4znpt 1/1 Running 0 12d
coredns-76f75df574-w8qh2 1/1 Running 0 12d
Step-by-Step Execution
Work in a dedicated namespace so policies are easy to reason about and cleanup is one command. Deploy web (NGINX), api (a simple HTTP responder), and db (PostgreSQL), each labelled so Services and NetworkPolicies can select them.
Create the namespace and workloads
kubectl create namespace netlab
kubectl -n netlab create deployment web --image=nginx:1.25 --port=80
kubectl -n netlab create deployment api --image=hashicorp/http-echo:0.2.3 -- -text="api-ok"
kubectl -n netlab create deployment db --image=postgres:16 --port=5432
kubectl -n netlab set env deployment/db POSTGRES_PASSWORD=labpass
Verify all pods are running
kubectl -n netlab get pods -o wide --show-labels
Expected output
NAME READY STATUS RESTARTS AGE IP LABELS
api-5f8c9d7b6d-k2j9x 1/1 Running 0 40s 10.244.0.12 app=api,...
db-6c8b5f7c9-4mztq 1/1 Running 0 40s 10.244.0.13 app=db,...
web-7d9c4b8f5-rp2lw 1/1 Running 0 40s 10.244.0.11 app=web,...
kubectl create deployment automatically sets app=<name>, and steps below rely on exactly those labels. If you build your own YAML later, keep app: web, app: api, and app: db consistent or your selectors will match nothing.What this teaches: The three-tier app gives every network primitive something to connect; deploy the workloads before the wiring.
Pod IPs are ephemeral, they change on every restart, reschedule, or scale event. A Service gives a stable virtual IP and DNS name that load-balances across matching pods. The default type, ClusterIP, is reachable only from inside the cluster, which is the correct default for anything that is not public-facing.
kubectl -n netlab expose deploy web --port 80 --target-port 80 --name webDo the same for api and db, then inspect what was created:
Expose remaining tiers and inspect the Services
kubectl -n netlab expose deployment api --port 8080 --target-port 5678
kubectl -n netlab expose deployment db --port 5432 --target-port 5432
kubectl -n netlab get svc
Expected output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api ClusterIP 10.96.214.87 <none> 8080/TCP 10s
db ClusterIP 10.96.108.201 <none> 5432/TCP 8s
web ClusterIP 10.96.45.173 <none> 80/TCP 2m
Prove internal reachability by curling the Service name (not the pod IP) from a throwaway pod. CoreDNS resolves web because the test pod lives in the same namespace:
Test Service DNS from inside the cluster
kubectl -n netlab run test --rm -it --image=busybox:1.36 -- wget -qO- http://web
Expected output (NGINX welcome page, truncated)
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
kubectl -n netlab describe svc web shows an Endpoints: list. If Endpoints is <none>, the Service selector does not match any pod labels, that mismatch, not DNS, is your problem.What this teaches: ClusterIP is the internal default; it proves pods can find each other without any external exposure.
Before going further, be deliberate about the exposure model, because it decides your attack surface:
- ClusterIP, internal only. Use for
apianddb; they should never be reachable from outside. - NodePort, opens a high port (30000-32767) on every node. Fine for quick demos, poor for production: no TLS, no host routing, and it bypasses your edge controls.
- Ingress, one external entry point that routes many hostnames and paths to many Services, terminates TLS, and centralizes logging and rate limiting. This is what you will use for
web.
You can see the NodePort mechanics without keeping it: patch the Service, observe the allocated port, then revert so the lab continues with only Ingress as the external path.
Peek at NodePort behaviour, then revert
kubectl -n netlab patch svc web -p '{"spec":{"type":"NodePort"}}'
kubectl -n netlab get svc web # note the 80:31xxx/TCP mapping
kubectl -n netlab patch svc web -p '{"spec":{"type":"ClusterIP"}}'
kubectl get svc -A | grep NodePort) is a real hygiene check on any cluster you inherit.What this teaches: Comparing the three types side by side makes the exposure ladder concrete: internal, node port, then L7 routing.
An Ingress resource is just a rule set; the controller you installed in setup turns it into live reverse-proxy config. This rule sends any request whose Host header is web.lab.tyfsadik.org to the web Service on port 80.
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: web }
spec:
ingressClassName: nginx
rules:
- host: web.lab.tyfsadik.org
http:
paths:
- path: /
pathType: Prefix
backend: { service: { name: web, port: { number: 80 } } }
Apply and confirm the rule is programmed
kubectl -n netlab apply -f ingress.yaml
kubectl -n netlab get ingress web
kubectl -n ingress-nginx get svc ingress-nginx-controller # grab the EXTERNAL-IP
Expected output
NAME CLASS HOSTS ADDRESS PORTS AGE
web nginx web.lab.tyfsadik.org 203.0.113.50 80 15s
Since the lab hostname is not in public DNS, send the Host header explicitly against the controller's external address (203.0.113.50 below is a documentation address, substitute your controller's actual EXTERNAL-IP):
Curl through the Ingress
curl -s -H 'Host: web.lab.tyfsadik.org' http://203.0.113.50/ | head -5
host: field, or the request hit a different controller. Confirm with kubectl -n netlab describe ingress web, the Events section shows whether the controller synced the rule.What this teaches: Ingress routes by host and path; one entry point can serve many backends, which is the production pattern.
Everything above worked because CoreDNS answers names like web, web.netlab, and the fully qualified web.netlab.svc.cluster.local. Take thirty seconds to see the record directly, you will need this reflex later when debugging egress policies, because a default-deny egress rule also blocks DNS if you forget to allow it.
Resolve the Service from inside a pod
kubectl -n netlab run dnscheck --rm -it --image=busybox:1.36 -- nslookup web.netlab.svc.cluster.local
Expected output
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: web.netlab.svc.cluster.local
Address 1: 10.96.45.173 web.netlab.svc.cluster.local
kube-dns ClusterIP (10.96.0.10 by default). If lookups hang, check kubectl -n kube-system logs deploy/coredns before suspecting your app.What this teaches: CoreDNS is the service discovery backbone; if names resolve, the platform's wiring is healthy.
Default-deny then explicit-allow is the zero-trust posture for pod traffic.
A NetworkPolicy with an empty podSelector selects every pod in the namespace, and an empty rule set means "allow nothing". The moment this applies, all ingress and egress in netlab is dropped, including DNS lookups from your pods, which is why step 07 adds DNS egress back immediately.
# default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: netlab
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
kubectl apply -f default-deny.yamlConfirm the policy exists and selects all pods
kubectl -n netlab get networkpolicy
kubectl -n netlab describe networkpolicy default-deny | head -12
Expected output
NAME POD-SELECTOR AGE
default-deny <none> 6s
POD-SELECTOR of <none> is correct here, it means "all pods", not "no pods". An empty podSelector: {} selects everything; forgetting the field entirely is invalid and the API will reject it.What this teaches: Default-deny flips the cluster from allow-all to deny-all; it is the moment network policy becomes real.
Now open exactly the paths the app needs: api pods may reach db on TCP 5432 (both the ingress half on db and the egress half on api, because default-deny blocks both directions), every pod may reach CoreDNS on UDP/TCP 53 so name resolution keeps working, and the ingress-nginx controller may reach web on port 80 so the external path from step 04 survives default-deny. Nothing else in the namespace can initiate anything.
# allow-api-to-db.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-to-db
namespace: netlab
spec:
podSelector:
matchLabels: { app: db }
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: { app: api }
ports:
- protocol: TCP
port: 5432
---
# allow-dns.yaml - egress to kube-dns so lookups survive default-deny
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: netlab
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# allow-api-egress-to-db.yaml, the egress half default-deny also blocks
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-egress-to-db
namespace: netlab
spec:
podSelector:
matchLabels: { app: api }
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels: { app: db }
ports:
- protocol: TCP
port: 5432
# allow-ingress-to-web.yaml, lets the ingress controller reach web under default-deny
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-web
namespace: netlab
spec:
podSelector:
matchLabels: { app: web }
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- protocol: TCP
port: 80
Apply the allow rules
kubectl apply -f allow-api-to-db.yaml
kubectl apply -f allow-api-egress-to-db.yaml
kubectl apply -f allow-dns.yaml
kubectl apply -f allow-ingress-to-web.yaml
kubectl -n netlab get networkpolicy
kubectl -n netlab get netpol -o wide.What this teaches: Each allow rule is a deliberate hole; writing them forces you to state what the app actually needs.
Segmentation that you have not tested is a hope, not a control. From a pod labelled app=other (which no policy permits), the database connection must fail; from an api pod it must succeed.
Launch an unauthorized pod and test the DB path
kubectl -n netlab run other --rm -it --image=busybox:1.36 --labels app=other -- nc -zv -w3 db 5432 || echo denied
kubectl -n netlab run api-probe --rm -i --image=busybox:1.36 --labels app=api -- nc -zv -w3 db 5432
exec into the api pod? hashicorp/http-echo is built from scratch: it has no shell and no nc. The probe pod carries the app=api label, so the policies treat it exactly like an api pod.Expected output
denied # from the 'other' pod (labeled app=other, which no policy selects)
db (10.96.108.201:5432) open # from an 'api' pod
The external path should still work end to end, because Ingress traffic enters through the controller in its own namespace and the web pod is only serving, not initiating:
Re-test the public path
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: web.lab.tyfsadik.org' http://203.0.113.50/
Expected output
200
nc from the probe pod hangs instead of printing open: the egress side is blocking it. The allow-api-to-db policy governs ingress to db, but default-deny also drops egress from api, which is why step 07 applies the separate allow-api-egress-to-db policy. If you skipped that file, apply it now. Symmetric allow rules are the most commonly forgotten half of a policy.What this teaches: Proving both the allow and the deny is the complete test; half-tested policy is assumed policy.
Step Flow Map
Each box is one step in the lab; work them in order, since later steps build on earlier state.
Progress So Far
How It Works
Every pod gets its own IP from the cluster's pod CIDR, and every Service gets a virtual ClusterIP that exists only as load-balancing rules. kube-proxy programs those rules into iptables or IPVS on every node: a packet sent to the ClusterIP is rewritten to one of the backend pod IPs. The Service IP is stable while pod IPs churn, which is why everything inside the cluster addresses services, never pods.
The Service types are a progression of exposure. ClusterIP is reachable only inside the cluster; NodePort opens the same port on every node's IP; LoadBalancer asks the cloud for an external IP; Ingress is different in kind, an L7 reverse proxy that routes by host and path to ClusterIP services. Comparing them side by side is what makes the tradeoffs concrete instead of theoretical.
CoreDNS resolves service names to ClusterIPs from inside any pod, which is the service discovery mechanism the whole platform leans on. NetworkPolicies are the firewall layer: by default all traffic is allowed, and a default-deny policy flips the cluster to deny-all so you can open only the paths the application needs. Policies are enforced by the CNI plugin, so the exact behavior depends on which CNI your cluster runs.
Testing & Validation
Run the full acceptance check in one pass: external reachability through the Ingress, and denial of an unauthorized internal path. On kind the controller Service never gets an EXTERNAL-IP (no cloud load balancer), so the check port-forwards the controller to localhost first.
# forward the ingress controller to localhost (kind has no LoadBalancer IP)
kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 &
PF=$!; sleep 3
curl -s -o /dev/null -w 'ingress HTTP code: %{http_code}\n' -H 'Host: web.lab.tyfsadik.org' http://localhost:8080/
kill $PF
# an unauthorized pod must NOT reach the database
kubectl -n netlab run other --rm -i --image=busybox:1.36 --labels app=other -- nc -zv -w3 db 5432 && echo 'POLICY BROKEN: db is reachable' || echo deniedThe curl must print ingress HTTP code: 200; anything else (000, 404, 503) means the Ingress rule, the host header, or the controller is misconfigured, stop and fix it before continuing. The probe pod must print denied; if it prints POLICY BROKEN the default-deny policy is not enforced on this cluster (usually the CNI, not the policy, is at fault on kind). Together the two checks confirm exposure plus segmentation.
For a final review, dump the effective policy set and the Ingress programming so you can read exactly what the cluster enforces:
Review the final state
kubectl -n netlab get svc,ingress,networkpolicy
kubectl -n netlab describe ingress web | tail -8
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | tail -5
Three green checks to look for: the Ingress has an ADDRESS, every NetworkPolicy lists the selectors you intended, and the controller access log shows your 200 for web.lab.tyfsadik.org. When you are done, kubectl delete namespace netlab tears down the entire lab, workloads, Services, and policies together.
Troubleshooting
Quick-reference for the failures you are most likely to hit in this lab.
| Symptom | Likely cause | Fix |
|---|---|---|
| Ingress returns 404 | Host or path rule does not match the request | Inspect the ingress rules and test with an explicit Host header via curl. |
| Service has no endpoints | The selector does not match any pod labels | An empty kubectl get endpoints plus kubectl get pods --show-labels reveals the mismatch. |
| DNS lookups fail from pods | CoreDNS is down or the query is malformed | Check the CoreDNS pods in kube-system, then test with a fully qualified name. |
| Default-deny broke everything | Deny applied without the allow rules the app needs | Add ingress and egress allows for DNS (port 53) first; even DNS needs an explicit hole. |
| NodePort unreachable from outside | Host firewall or the port mapping blocks it | Verify the node port with kubectl get svc and check the host firewall rules. |
Troubleshooting
- Ingress 404: the host/path rule must match the request Host header. Verify with
kubectl -n netlab describe ingress weband confirm you are hitting the right controller address. A missingingressClassNameon clusters with multiple controllers causes the same symptom. - Policy has no effect: your CNI must support NetworkPolicy (Calico/Cilium), not all do. kind's default kindnet and some managed-node images ignore policies entirely, traffic keeps flowing and nothing logs why. Check
kubectl -n kube-system get pods | grep -Ei 'calico|cilium'. - DNS fails inside pods: check CoreDNS pods are Running in kube-system. If DNS broke only after applying default-deny, your egress policy is blocking port 53, apply the
allow-dnspolicy from step 07. - Ingress EXTERNAL-IP stuck at
<pending>: there is no cloud load balancer to allocate one. On minikube runminikube tunnelin a second terminal; elsewhere, port-forward directly withkubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80and test againsthttp://localhost:8080. - Everything in the namespace broke after default-deny: that is the policy working, not failing. Rebuild connectivity one allow rule at a time, DNS egress first, then app-to-app paths, and test after each so you know which rule restored which flow.
Extension Ideas
- Add TLS to the Ingress with cert-manager: install the CRDs, create a ClusterIssuer against Let's Encrypt staging, and annotate the Ingress with
cert-manager.io/cluster-issuerso the certificate is issued and rotated automatically. - Write an egress policy that allows
apito reach only one external API endpoint by CIDR, and observe how ipBlock rules interact with DNS resolution. - Move to eBPF policy in the Zero-Trust Hybrid Cloud lab.
- Deploy this on managed Kubernetes in AKS.
Key Results
- Exposed an app externally through an Ingress controller.
- Enforced default-deny pod traffic with explicit allow rules.
- Verified both external reachability and internal segmentation.
- Applied a zero-trust posture inside the cluster.
Key Takeaways
- Services give stable IPs and DNS names over ephemeral pods; ClusterIP is the safe default, NodePort and LoadBalancer widen the attack surface.
- An Ingress is only a rule set, the controller does the work, and the Host header must match the rule exactly.
- NetworkPolicy is enforced by the CNI, not by Kubernetes itself; without a policy-capable CNI it fails silently.
- Default-deny first, then additive allow rules, and remember egress covers DNS too, or everything appears "down".
- Always test both sides: an allowed path that works and a denied path that fails are the proof the segmentation is real.
Practice Challenges
Challenge 1: Two hosts, one ingress
Goal: Add a second ingress host routing to a different backend service.
Hint: One Ingress object can hold many rules; the host field is the router.
Challenge 2: Least-privilege policy
Goal: Write a NetworkPolicy allowing only frontend to backend on 5432, then prove the database rejects anything else.
Hint: Test from a temporary debug pod; its blocked connection is the proof.
Challenge 3: Compare kube-proxy modes
Goal: Check whether your cluster uses iptables or IPVS mode and inspect the programmed rules.
Hint: The kube-proxy pod logs name the mode at startup.