You will expose an app through an Ingress and lock down pod-to-pod traffic with a default-deny NetworkPolicy. By the end external users reach your service while only authorized pods can talk to the backend.

Kubernetes networking is flat by design: every pod can reach every other pod on every node, with no NAT and no built-in firewall. That is great for developer velocity and terrible for security. This lab walks the full lifecycle of getting traffic into the cluster safely (Services and Ingress) and then constraining how traffic moves inside it (NetworkPolicy). You will work with a three-tier demo app, a public web frontend, an internal api, and a db backend, so every rule you write maps to a real allow-list decision.

Learning Objectives

  • Compare ClusterIP, NodePort, and Ingress for exposure.
  • Route external traffic via an Ingress controller.
  • Enforce default-deny with NetworkPolicy.
  • Verify service discovery through CoreDNS from inside a pod.
  • Prove segmentation works by testing both allowed and denied paths.
  • Time: ~4 hours · Difficulty: Advanced · Prereqs: a cluster with an Ingress controller and a CNI that supports policy.

Architecture Overview

Traffic enters through the Ingress controller, which terminates the HTTP request and routes it by host and path to a Service. The Service load-balances across the api pods. Those pods are the only workloads allowed to reach the db pods, because a default-deny NetworkPolicy blocks everything else, including pods in the same namespace that have no business touching the database.

Three layers do the work here. kube-proxy programs iptables or IPVS rules so a stable Service virtual IP forwards to healthy pod endpoints. The Ingress controller (a reverse proxy such as NGINX running as a pod) watches Ingress resources and rewrites its own config to match. The CNI plugin (Calico or Cilium) enforces NetworkPolicy at the pod's virtual interface, dropping packets that match no allow rule. NetworkPolicy itself is only a declaration, without a policy-capable CNI it is silently ignored.

Environment Setup

Prerequisites

  • A local cluster (kind) with kubectl, plus an ingress controller installed.
  • The three-tier demo manifests from the lab, or any frontend, backend, and database set.
  • Assumed skills: Kubernetes basics (pods, deployments, services) and DNS fundamentals.

You will need: a cluster, an Ingress controller (ingress-nginx), and a policy-capable CNI (Calico/Cilium).

Before you begin: the default cluster allows all pod traffic until you add a policy.

First confirm your context points at the right cluster and that every node is Ready. A NotReady node will silently swallow pod scheduling later and waste your debugging time.

Check cluster access and node health

kubectl cluster-info
kubectl get nodes -o wide

Output:

$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:39969
CoreDNS is running at https://127.0.0.1:39969/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get nodes -o wide
NAME                STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION              CONTAINER-RUNTIME
lab-control-plane   Ready    control-plane   61s   v1.37.0   172.19.0.2    <none>        Debian GNU/Linux 13 (trixie)   6.8.0-117-generic (arm64)   containerd://2.3.4

Next, verify the CNI actually enforces policy. Look for Calico or Cilium pods in kube-system. If you are on kind or Docker Desktop with the default CNI (kindnet), NetworkPolicy will apply without error but do nothing, the classic silent failure of this lab.

Confirm a policy-capable CNI is running

kubectl -n kube-system get pods | grep -Ei 'calico|cilium'

Output:

$ kubectl -n kube-system get pods | grep -Ei 'calico|cilium'
calico-kube-controllers-db57f7644-dnwzs     1/1     Running   0          52s
calico-node-mfbs4                           1/1     Running   0          52s
How to verify: no output from that grep means no policy enforcement. On a fresh cluster, install Calico with kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml before continuing.
Do not install Calico over kind's default CNI. kind ships with kindnet, and applying the Calico manifest on top leaves two CNIs fighting over pod networking. Create the kind cluster with the default CNI disabled first:
# kind-calico.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: 192.168.0.0/16
kind create cluster --name lab --config kind-calico.yaml
Then install Calico on the empty CNI slot. The podSubnet must match the IP pool in the Calico manifest (192.168.0.0/16 by default); a mismatch leaves nodes without usable pod routes. Skipping the CNI swap is the most common reason policies silently do nothing on kind.

Install the ingress-nginx controller if it is not present. The manifest creates the ingress-nginx namespace, the controller Deployment, and a Service of type LoadBalancer that fronts it.

Retirement notice: the ingress-nginx project was archived by its maintainers on 2026-03-24 and is now read-only; controller-v1.15.1 is the final release and will receive no security patches. It still works for this lab, but plan new deployments on the Gateway API (for example Envoy Gateway) instead of new ingress-nginx installs.

Install ingress-nginx and wait for the controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/cloud/deploy.yaml
kubectl -n ingress-nginx wait --for=condition=ready pod -l app.kubernetes.io/component=controller --timeout=120s

On minikube use minikube addons enable ingress instead. On bare metal or kind with no cloud load balancer, the controller Service will sit at <pending> for EXTERNAL-IP, see the Troubleshooting section for the port-forward workaround.

Finally, sanity-check CoreDNS, because every Service lookup in this lab depends on it:

Verify CoreDNS is healthy

kubectl -n kube-system get pods -l k8s-app=kube-dns

Output:

$ kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-559f6c778d-dvlcn   1/1     Running   0          52s
coredns-559f6c778d-sxfjd   1/1     Running   0          52s

Step-by-Step Execution

The outputs on this page come from a re-run on 14 November 2026 in an Ubuntu 24.04 container running its own Docker Engine and a kind cluster with Calico and ingress-nginx (arm64). Sample files, users, and hosts were created for the run, so names, numbers, and timestamps reflect that run. Steps that could not be re-run there are marked as example output, with the reason.

01
Deploy the three-tier demo app

Work in a dedicated namespace so policies are easy to reason about and cleanup is one command. Deploy web (NGINX), api (a simple HTTP responder), and db (PostgreSQL), each labelled so Services and NetworkPolicies can select them.

Create the namespace and workloads

kubectl create namespace netlab
kubectl -n netlab create deployment web --image=nginx:1.25 --port=80
kubectl -n netlab create deployment api --image=hashicorp/http-echo:1.0 -- /http-echo -text="api-ok"
kubectl -n netlab create deployment db --image=postgres:16 --port=5432
kubectl -n netlab set env deployment/db POSTGRES_PASSWORD=labpass

Verify all pods are running

kubectl -n netlab get pods -o wide --show-labels

Output:

$ kubectl -n netlab get pods -o wide --show-labels
NAME                   READY   STATUS    RESTARTS   AGE   IP                NODE                NOMINATED NODE   READINESS GATES   LABELS
api-6cbd5599b7-tm6cz   1/1     Running   0          46s   192.168.180.138   lab-control-plane   <none>           <none>            app=api,pod-template-hash=6cbd5599b7
db-54bf98ff4f-8gh7h    1/1     Running   0          45s   192.168.180.139   lab-control-plane   <none>           <none>            app=db,pod-template-hash=54bf98ff4f
web-5dd8cd7b9c-6zsfl   1/1     Running   0          46s   192.168.180.136   lab-control-plane   <none>           <none>            app=web,pod-template-hash=5dd8cd7b9c
Common mistake: forgetting the label. Every kubectl create deployment automatically sets app=<name>, and steps below rely on exactly those labels. If you build your own YAML later, keep app: web, app: api, and app: db consistent or your selectors will match nothing.

What this teaches: The three-tier app gives every network primitive something to connect; deploy the workloads before the wiring.

02
Expose a service internally

Pod IPs are ephemeral, they change on every restart, reschedule, or scale event. A Service gives a stable virtual IP and DNS name that load-balances across matching pods. The default type, ClusterIP, is reachable only from inside the cluster, which is the correct default for anything that is not public-facing.

kubectl -n netlab expose deploy web --port 80 --target-port 80 --name web

Do the same for api and db, then inspect what was created:

Expose remaining tiers and inspect the Services

kubectl -n netlab expose deployment api --port 8080 --target-port 5678
kubectl -n netlab expose deployment db --port 5432 --target-port 5432
kubectl -n netlab get svc

Output:

$ kubectl -n netlab expose deployment api --port 8080 --target-port 5678
service/api exposed
$ kubectl -n netlab expose deployment db --port 5432 --target-port 5432
service/db exposed
$ kubectl -n netlab get svc
NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
api    ClusterIP   10.96.96.158    <none>        8080/TCP   0s
db     ClusterIP   10.96.120.44    <none>        5432/TCP   0s
web    ClusterIP   10.96.243.111   <none>        80/TCP     0s

Prove internal reachability by curling the Service name (not the pod IP) from a throwaway pod. CoreDNS resolves web because the test pod lives in the same namespace:

Test Service DNS from inside the cluster

kubectl -n netlab run test --rm -it --image=busybox:1.36 -- wget -qO- http://web

Output (the nginx welcome page, first 4 lines):

$ kubectl -n netlab run test --rm -it --image=busybox:1.36 -- wget -qO- http://web
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
How to verify: kubectl -n netlab describe svc web shows an Endpoints: list. If Endpoints is <none>, the Service selector does not match any pod labels, that mismatch, not DNS, is your problem.

What this teaches: ClusterIP is the internal default; it proves pods can find each other without any external exposure.

03
Compare ClusterIP, NodePort, and Ingress

Before going further, be deliberate about the exposure model, because it decides your attack surface:

  • ClusterIP, internal only. Use for api and db; they should never be reachable from outside.
  • NodePort, opens a high port (30000-32767) on every node. Fine for quick demos, poor for production: no TLS, no host routing, and it bypasses your edge controls.
  • Ingress, one external entry point that routes many hostnames and paths to many Services, terminates TLS, and centralizes logging and rate limiting. This is what you will use for web.

You can see the NodePort mechanics without keeping it: patch the Service, observe the allocated port, then revert so the lab continues with only Ingress as the external path.

Peek at NodePort behaviour, then revert

kubectl -n netlab patch svc web -p '{"spec":{"type":"NodePort"}}'
kubectl -n netlab get svc web   # note the 80:31xxx/TCP mapping
kubectl -n netlab patch svc web -p '{"spec":{"type":"ClusterIP"}}'
Why this matters for security: every NodePort is an unauthenticated listener on every node IP. Auditing for stray NodePorts (kubectl get svc -A | grep NodePort) is a real hygiene check on any cluster you inherit.

What this teaches: Comparing the three types side by side makes the exposure ladder concrete: internal, node port, then L7 routing.

04
Route external traffic with Ingress

An Ingress resource is just a rule set; the controller you installed in setup turns it into live reverse-proxy config. This rule sends any request whose Host header is web.lab.tyfsadik.org to the web Service on port 80.

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: web }
spec:
  ingressClassName: nginx
  rules:
    - host: web.lab.tyfsadik.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: web, port: { number: 80 } } }

Apply and confirm the rule is programmed

kubectl -n netlab apply -f ingress.yaml
kubectl -n netlab get ingress web
kubectl -n ingress-nginx get svc ingress-nginx-controller   # grab the EXTERNAL-IP

Output:

$ kubectl -n netlab apply -f ingress.yaml
ingress.networking.k8s.io/web created
$ kubectl -n netlab get ingress web
NAME   CLASS   HOSTS                  ADDRESS   PORTS   AGE
web    nginx   web.lab.tyfsadik.org             80      0s
$ kubectl -n ingress-nginx get svc ingress-nginx-controller   # grab the EXTERNAL-IP
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.96.165.79   <pending>     80:31518/TCP,443:32591/TCP   75s

Since the lab hostname is not in public DNS, send the Host header explicitly against the controller's external address (203.0.113.50 below is a documentation address, substitute your controller's actual EXTERNAL-IP):

Curl through the Ingress

curl -s -H 'Host: web.lab.tyfsadik.org' http://203.0.113.50/ | head -5
Common mistake: a 404 here almost always means the Host header does not match the host: field, or the request hit a different controller. Confirm with kubectl -n netlab describe ingress web, the Events section shows whether the controller synced the rule.

What this teaches: Ingress routes by host and path; one entry point can serve many backends, which is the production pattern.

05
Verify CoreDNS service discovery

Everything above worked because CoreDNS answers names like web, web.netlab, and the fully qualified web.netlab.svc.cluster.local. Take thirty seconds to see the record directly, you will need this reflex later when debugging egress policies, because a default-deny egress rule also blocks DNS if you forget to allow it.

Resolve the Service from inside a pod

kubectl -n netlab run dnscheck --rm -it --image=busybox:1.36 -- nslookup web.netlab.svc.cluster.local

Output:

$ kubectl -n netlab run dnscheck --rm -it --image=busybox:1.36 -- nslookup web.netlab.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	web.netlab.svc.cluster.local
Address: 10.96.243.111

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/dnscheck, falling back to streaming logs: Internal error occurred: Internal error occurred: error attaching to container: failed to load task: no running task found: task 3f685d0be372449a49b1358b8fa1473e687e2f0a1a13e677ffb0336a5736176c not found
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	web.netlab.svc.cluster.local
Address: 10.96.243.111

pod "dnscheck" deleted from netlab namespace
How to verify: the answering server should be the kube-dns ClusterIP (10.96.0.10 by default). If lookups hang, check kubectl -n kube-system logs deploy/coredns before suspecting your app.

What this teaches: CoreDNS is the service discovery backbone; if names resolve, the platform's wiring is healthy.

06
Apply a default-deny policy

Default-deny then explicit-allow is the zero-trust posture for pod traffic.

A NetworkPolicy with an empty podSelector selects every pod in the namespace, and an empty rule set means "allow nothing". The moment this applies, all ingress and egress in netlab is dropped, including DNS lookups from your pods, which is why step 07 adds DNS egress back immediately.

# default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: netlab
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
kubectl apply -f default-deny.yaml

Confirm the policy exists and selects all pods

kubectl -n netlab get networkpolicy
kubectl -n netlab describe networkpolicy default-deny | head -12

Output:

$ kubectl -n netlab get networkpolicy
NAME           POD-SELECTOR   AGE
default-deny   <none>         0s
$ kubectl -n netlab describe networkpolicy default-deny | head -12
Name:         default-deny
Namespace:    netlab
Created on:   2026-11-14 21:15:12 -0500 EST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Ingress, Egress
Common mistake: a POD-SELECTOR of <none> is correct here, it means "all pods", not "no pods". An empty podSelector: {} selects everything; forgetting the field entirely is invalid and the API will reject it.

What this teaches: Default-deny flips the cluster from allow-all to deny-all; it is the moment network policy becomes real.

07
Poke only the holes you need

Now open exactly the paths the app needs: api pods may reach db on TCP 5432 (both the ingress half on db and the egress half on api, because default-deny blocks both directions), every pod may reach CoreDNS on UDP/TCP 53 so name resolution keeps working, and the ingress-nginx controller may reach web on port 80 so the external path from step 04 survives default-deny. Nothing else in the namespace can initiate anything.

# allow-api-to-db.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: netlab
spec:
  podSelector:
    matchLabels: { app: db }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: api }
      ports:
        - protocol: TCP
          port: 5432
---
# allow-dns.yaml - egress to kube-dns so lookups survive default-deny
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: netlab
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# allow-api-egress-to-db.yaml, the egress half default-deny also blocks

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-egress-to-db
  namespace: netlab
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: db }
      ports:
        - protocol: TCP
          port: 5432

# allow-ingress-to-web.yaml, lets the ingress controller reach web under default-deny

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-web
  namespace: netlab
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 80

Apply the allow rules

kubectl apply -f allow-api-to-db.yaml
kubectl apply -f allow-api-egress-to-db.yaml
kubectl apply -f allow-dns.yaml
kubectl apply -f allow-ingress-to-web.yaml
kubectl -n netlab get networkpolicy
How to verify: rules are additive, the union of all policies selecting a pod is what is allowed. There is no "deny" statement and no rule ordering. If traffic flows that you expected to be blocked, some other policy is allowing it; list them all with kubectl -n netlab get netpol -o wide.

What this teaches: Each allow rule is a deliberate hole; writing them forces you to state what the app actually needs.

08
Prove both the allow and the deny

Segmentation that you have not tested is a hope, not a control. From a pod labelled app=other (which no policy permits), the database connection must fail; from an api pod it must succeed.

Launch an unauthorized pod and test the DB path

kubectl -n netlab run other --rm -i --image=busybox:1.36 --labels app=other -- sh -c 'nc -zv -w3 db 5432 && echo reachable || echo denied'
kubectl -n netlab run api-probe --rm -i --image=busybox:1.36 --labels app=api -- nc -zv -w3 db 5432
Why a throwaway pod instead of exec into the api pod? hashicorp/http-echo is built from scratch: it has no shell and no nc. The probe pod carries the app=api label, so the policies treat it exactly like an api pod.

Output:

$ kubectl -n netlab run other --rm -i --image=busybox:1.36 --labels app=other -- sh -c 'nc -zv -w3 db 5432 && echo reachable || echo denied'
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
nc: db (10.96.120.44:5432): Connection timed out
denied
pod "other" deleted from netlab namespace
$ kubectl -n netlab run api-probe --rm -i --image=busybox:1.36 --labels app=api -- nc -zv -w3 db 5432
db (10.96.120.44:5432) open
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/api-probe, falling back to streaming logs: unable to upgrade connection: container api-probe not found in pod api-probe_netlab
db (10.96.120.44:5432) open
pod "api-probe" deleted from netlab namespace

The external path should still work end to end, because Ingress traffic enters through the controller in its own namespace and the web pod is only serving, not initiating:

Re-test the public path

curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: web.lab.tyfsadik.org' http://203.0.113.50/

Example output (not re-run: it curls the ingress controller's cloud LoadBalancer address, which kind does not provide):

200
If nc from the probe pod hangs instead of printing open: the egress side is blocking it. The allow-api-to-db policy governs ingress to db, but default-deny also drops egress from api, which is why step 07 applies the separate allow-api-egress-to-db policy. If you skipped that file, apply it now. Symmetric allow rules are the most commonly forgotten half of a policy.

What this teaches: Proving both the allow and the deny is the complete test; half-tested policy is assumed policy.

Step Flow Map

Each box is one step in the lab; work them in order, since later steps build on earlier state.

Progress So Far

How It Works

Every pod gets its own IP from the cluster's pod CIDR, and every Service gets a virtual ClusterIP that exists only as load-balancing rules. kube-proxy programs those rules into iptables or IPVS on every node: a packet sent to the ClusterIP is rewritten to one of the backend pod IPs. The Service IP is stable while pod IPs churn, which is why everything inside the cluster addresses services, never pods.

The Service types are a progression of exposure. ClusterIP is reachable only inside the cluster; NodePort opens the same port on every node's IP; LoadBalancer asks the cloud for an external IP; Ingress is different in kind, an L7 reverse proxy that routes by host and path to ClusterIP services. Comparing them side by side is what makes the tradeoffs concrete instead of theoretical.

CoreDNS resolves service names to ClusterIPs from inside any pod, which is the service discovery mechanism the whole platform leans on. NetworkPolicies are the firewall layer: by default all traffic is allowed, and a default-deny policy flips the cluster to deny-all so you can open only the paths the application needs. Policies are enforced by the CNI plugin, so the exact behavior depends on which CNI your cluster runs.

Testing & Validation

Run the full acceptance check in one pass: external reachability through the Ingress, and denial of an unauthorized internal path. On kind the controller Service never gets an EXTERNAL-IP (no cloud load balancer), so the check port-forwards the controller to localhost first.

# forward the ingress controller to localhost (kind has no LoadBalancer IP) kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 & PF=$!; sleep 3 curl -s -o /dev/null -w 'ingress HTTP code: %{http_code}\n' -H 'Host: web.lab.tyfsadik.org' http://localhost:8080/ kill $PF # an unauthorized pod must NOT reach the database kubectl -n netlab run other --rm -i --image=busybox:1.36 --labels app=other -- sh -c 'nc -zv -w3 db 5432 && echo "POLICY BROKEN: db is reachable" || echo denied'

The curl must print ingress HTTP code: 200; anything else (000, 404, 503) means the Ingress rule, the host header, or the controller is misconfigured, stop and fix it before continuing. The probe pod must print denied; if it prints POLICY BROKEN the default-deny policy is not enforced on this cluster (usually the CNI, not the policy, is at fault on kind). Together the two checks confirm exposure plus segmentation. The probe prints its verdict from inside the pod (sh -c 'nc ... && echo ... || echo denied') because kubectl run --rm exits 0 even when the pod's command fails, so an || outside the pod never fires, and an && echo 'POLICY BROKEN' outside it would report a broken policy on a working one.

For a final review, dump the effective policy set and the Ingress programming so you can read exactly what the cluster enforces:

Review the final state

kubectl -n netlab get svc,ingress,networkpolicy
kubectl -n netlab describe ingress web | tail -8
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | tail -5

Three green checks to look for: the Ingress has an ADDRESS, every NetworkPolicy lists the selectors you intended, and the controller access log shows your 200 for web.lab.tyfsadik.org. When you are done, kubectl delete namespace netlab tears down the entire lab, workloads, Services, and policies together.

Troubleshooting

Quick-reference for the failures you are most likely to hit in this lab.

SymptomLikely causeFix
Ingress returns 404Host or path rule does not match the requestInspect the ingress rules and test with an explicit Host header via curl.
Service has no endpointsThe selector does not match any pod labelsAn empty kubectl get endpoints plus kubectl get pods --show-labels reveals the mismatch.
DNS lookups fail from podsCoreDNS is down or the query is malformedCheck the CoreDNS pods in kube-system, then test with a fully qualified name.
Default-deny broke everythingDeny applied without the allow rules the app needsAdd ingress and egress allows for DNS (port 53) first; even DNS needs an explicit hole.
NodePort unreachable from outsideHost firewall or the port mapping blocks itVerify the node port with kubectl get svc and check the host firewall rules.
Troubleshooting
  • Ingress 404: the host/path rule must match the request Host header. Verify with kubectl -n netlab describe ingress web and confirm you are hitting the right controller address. A missing ingressClassName on clusters with multiple controllers causes the same symptom.
  • Policy has no effect: your CNI must support NetworkPolicy (Calico/Cilium), not all do. kind's default kindnet and some managed-node images ignore policies entirely, traffic keeps flowing and nothing logs why. Check kubectl -n kube-system get pods | grep -Ei 'calico|cilium'.
  • DNS fails inside pods: check CoreDNS pods are Running in kube-system. If DNS broke only after applying default-deny, your egress policy is blocking port 53, apply the allow-dns policy from step 07.
  • Ingress EXTERNAL-IP stuck at <pending>: there is no cloud load balancer to allocate one. On minikube run minikube tunnel in a second terminal; elsewhere, port-forward directly with kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 and test against http://localhost:8080.
  • Everything in the namespace broke after default-deny: that is the policy working, not failing. Rebuild connectivity one allow rule at a time, DNS egress first, then app-to-app paths, and test after each so you know which rule restored which flow.

Extension Ideas

  • Add TLS to the Ingress with cert-manager: install the CRDs, create a ClusterIssuer against Let's Encrypt staging, and annotate the Ingress with cert-manager.io/cluster-issuer so the certificate is issued and rotated automatically.
  • Write an egress policy that allows api to reach only one external API endpoint by CIDR, and observe how ipBlock rules interact with DNS resolution.
  • Move to eBPF policy in the Zero-Trust Hybrid Cloud lab.
  • Deploy this on managed Kubernetes in AKS.

Key Results

  • Exposed an app externally through an Ingress controller.
  • Enforced default-deny pod traffic with explicit allow rules.
  • Verified both external reachability and internal segmentation.
  • Applied a zero-trust posture inside the cluster.

Key Takeaways

  • Services give stable IPs and DNS names over ephemeral pods; ClusterIP is the safe default, NodePort and LoadBalancer widen the attack surface.
  • An Ingress is only a rule set, the controller does the work, and the Host header must match the rule exactly.
  • NetworkPolicy is enforced by the CNI, not by Kubernetes itself; without a policy-capable CNI it fails silently.
  • Default-deny first, then additive allow rules, and remember egress covers DNS too, or everything appears "down".
  • Always test both sides: an allowed path that works and a denied path that fails are the proof the segmentation is real.

Practice Challenges

Challenge 1: Two hosts, one ingress

Goal: Add a second ingress host routing to a different backend service.

Hint: One Ingress object can hold many rules; the host field is the router.

Challenge 2: Least-privilege policy

Goal: Write a NetworkPolicy allowing only frontend to backend on 5432, then prove the database rejects anything else.

Hint: Test from a temporary debug pod; its blocked connection is the proof.

Challenge 3: Compare kube-proxy modes

Goal: Check whether your cluster uses iptables or IPVS mode and inspect the programmed rules.

Hint: The kube-proxy pod logs name the mode at startup.

Frequently Asked Questions

How do I fix 'Policy has no effect'?

your CNI must support NetworkPolicy (Calico/Cilium), not all do. kind's default kindnet and some managed-node images ignore policies entirely, traffic keeps flowing and nothing logs why. Check kubectl -n kube-system get pods | grep -Ei 'calico|cilium'.

How do I fix 'DNS fails inside pods'?

check CoreDNS pods are Running in kube-system. If DNS broke only after applying default-deny, your egress policy is blocking port 53, apply the allow-dns policy from step 07.

How do I fix 'Ingress EXTERNAL-IP stuck at <pending>'?

there is no cloud load balancer to allocate one. On minikube run minikube tunnel in a second terminal; elsewhere, port-forward directly with kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 and test against http://localhost:8080.

How do I fix 'Everything in the namespace broke after default-deny'?

that is the policy working, not failing. Rebuild connectivity one allow rule at a time, DNS egress first, then app-to-app paths, and test after each so you know which rule restored which flow.

Cite This Page

Citation formats

Plain text

Taki Sadik, "Kubernetes Networking: Services, Ingress, and NetworkPolicy", TYFSADIK, https://tyfsadik.org/blog/labs/containers-kubernetes/kubernetes-networking (accessed 2026-10-22).

BibTeX

@misc{sadik-kubernetes-networking,
  author  = {Sadik, MD. Taki Yasir Faraji},
  title   = {Kubernetes Networking: Services, Ingress, and NetworkPolicy},
  year    = {2026},
  url     = {https://tyfsadik.org/blog/labs/containers-kubernetes/kubernetes-networking},
  urldate = {2026-10-22}
}

APA

Sadik, M. T. Y. F. (2026). Kubernetes Networking: Services, Ingress, and NetworkPolicy. TYFSADIK. https://tyfsadik.org/blog/labs/containers-kubernetes/kubernetes-networking