Deploy to Azure Kubernetes Service (AKS) with Autoscaling

This is the only heavy Azure operation in the lab. One az aks create call provisions the managed control plane and a system node pool backed by a Virtual Machine Scale Set. The flags matter: --enable-cluster-autoscaler turns on node-level autoscaling from day one (it is painful to retrofit), --min-count 2 keeps two nodes for redundancy, and --max-count 5 caps your spend. --generate-ssh-keys creates a keypair installed on the nodes for debugging (for example with az aks node-debug). It does not make the nodes publicly reachable: AKS nodes get no public IPs by default, so the keys alone do not give you SSH access from the internet. Expect provisioning to take roughly 5 to 10 minutes.

Verify the cluster finished provisioning and note the Kubernetes version Azure picked (the 1.31.7 below is illustrative; Azure selects the current default at creation time):

Common mistake: re-running az aks create after a failure without reading the error, if the error mentions quota (e.g. Operation could not be completed as it results in exceeding approved Total Regional Cores quota), no retry will fix it; switch VM size or region. Also note kubectl's long flags need two dashes (--node-count, not -node-count), a single dash is parsed as bundled short flags and fails.

Expected output:

$ az aks show -g rg -n tyf-aks --query "{state:provisioningState, k8s:kubernetesVersion}" -o table
State      K8s
---------  -------
Succeeded  1.31.7

What this teaches: Autoscaling node pools let the cluster grow with demand; the min/max bounds are the budget guardrail.

az aks get-credentials fetches the cluster's kubeconfig from Azure and merges it into ~/.kube/config, setting AKS as your current context. Without this step kubectl has no idea where your API server is, every command would target localhost:8080 and fail with a connection-refused error. The second half of the command creates a Deployment: one declarative object that keeps a replica set of nginx pods alive.

Watch the pod come up, it should pass through ContainerCreating to Running within a minute or two (the first image pull takes the longest):

Common mistake: running kubectl against the wrong context after working with other clusters (minikube, Docker Desktop). Check kubectl config current-context, it must say tyf-aks. If you rebuilt the cluster, re-run az aks get-credentials --overwrite-existing so stale certificates do not linger.

Expected output:

Merged "tyf-aks" as current context in /home/taki/.kube/config

What this teaches: Merging kubeconfig is what connects kubectl to the managed control plane; verify the context before deploying.

Two things happen here. kubectl expose ... --type=LoadBalancer creates a Service and, because this is AKS, the cloud-controller-manager immediately provisions a real Azure Load Balancer with a public IP in front of your pods. kubectl autoscale then creates an HPA that targets 60% average CPU utilization across pods, bounded between 2 and 10 replicas.

One critical detail most quickstarts skip: CPU-based HPA utilization is computed as a percentage of the pod's CPU request, and a Deployment created imperatively has no requests set. Without them the HPA cannot compute a percentage and reports <unknown> forever. Set requests (and a limit for good hygiene) explicitly:

SET CPU REQUESTS SO THE HPA CAN MEASURE

kubectl set resources deployment web \
  --requests=cpu=250m --limits=cpu=500m

Expected output:

$ kubectl get hpa web
NAME   REFERENCE        TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   3%/60%    2         10        2          2m

How to verify: TARGETS must show current/target as real percentages (e.g. 3%/60%). If it shows <unknown>/60% for more than two or three minutes, either the resource requests are missing (fix above) or the metrics-server is not answering, see Troubleshooting. Note the HPA already bumped the Deployment to 2 replicas, its floor.

What this teaches: The HPA needs resource requests to compute utilization; without the denominator, autoscaling cannot work.

Azure takes a minute or two to allocate the public IP and program the load-balancer rules. Until then the Service shows <pending>. Watch the Service object until EXTERNAL-IP appears, then curl it.

That title from the default nginx page, served from a pod on an AKS node through an Azure Load Balancer, is your end-to-end proof: deployment, service, cloud networking, and kube-proxy routing all work. The CLUSTER-IP is internal-only; the EXTERNAL-IP is the one the world can reach.

Common mistake: curling the ClusterIP from your laptop, it is only routable inside the cluster's virtual network. Always use the EXTERNAL-IP. If it stays pending beyond five minutes, run kubectl describe svc web and read the Events section at the bottom.

Expected output:

$ kubectl get svc web -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
20.81.44.12

What this teaches: A public IP on the service is the moment the app is truly reachable; test it from outside the cluster.

Now the fun part: make the HPA react. Run a throwaway busybox pod that hammers the Service in a loop, because it resolves the Service name web over cluster DNS, the load spreads across all replicas through the internal ClusterIP. The nginx pods' CPU climbs, the metrics-server reports it, and the HPA starts adding replicas to bring average utilization back under 60%.

LOAD GENERATOR (LEAVE RUNNING IN THIS TERMINAL)

kubectl run -i --tty loadgen --rm --restart=Never \
  --image=busybox:1.36 -- /bin/sh -c \
  "for i in 1 2 3 4; do while true; do wget -q -O- http://web > /dev/null; done & done; wait"

Expected output:

In a second terminal, watch the HPA. Within a couple of minutes you should see current CPU blow past the target and the replica count climb:

The math is simple: at 145% of the 60% target the HPA computes ceil(2 × 145/60) = 5 replicas and patches the Deployment. New pods schedule onto the nodes, pull the image, and start absorbing traffic, average utilization falls back toward the target. The feedback loop is closed.

Common mistake: giving up after 30 seconds. The HPA polls metrics every ~15 seconds and has a stabilization window; allow 2 to 3 minutes for the first scale-up and 5+ minutes for scale-down after you stop the load (Ctrl+C the loadgen pod to stop it, scale-down is deliberately slower to avoid flapping).

What this teaches: Load generation is how you prove the HPA reacts; watch the replica count climb with the metric.

Pod-level scaling only works while the nodes have room. Each web pod requests 250m CPU. On the default 2-vCPU node size, 10 replicas request 2500m against roughly 3800m allocatable across two nodes, so as the HPA pushes toward 8 to 10 replicas the nodes fill up and new pods sit Pending with Insufficient cpu. If you created the cluster with a larger VM size, 10 small pods may still fit: raise the HPA --max or add load until pods genuinely do not fit, the autoscaler only reacts to pods that cannot schedule. That is exactly the signal the cluster autoscaler listens for: it grows the VM Scale Set from 2 toward the 5-node ceiling so the pending pods can schedule.

The web pods alone may or may not exhaust the nodes, depending on the VM size you picked. To make the Pending state deterministic, add a dedicated pressure workload: twenty pods each requesting 500m CPU (10 CPU total). Every single pod fits on any node size, but twenty of them cannot fit on two nodes of any reasonable size, so some are guaranteed to sit Pending and the autoscaler must react. Delete it after the demo so it does not fight the HPA for room:

GUARANTEE PENDING PODS WITH A PRESSURE WORKLOAD

kubectl create deployment pressure --image=nginx:stable --replicas=20
kubectl set resources deployment pressure --requests=cpu=500m
kubectl get pods -l app=pressure | grep -c Pending   # expect a nonzero count within a minute
# after you see the new node join: kubectl delete deployment pressure

The third node appearing with a tiny AGE is the autoscaler doing its job, from Pending pod to Ready node typically takes 3 to 5 minutes (VM provisioning plus kubelet registration). You can corroborate it from the Azure side with az aks show -g rg -n tyf-aks --query properties.agentPoolProfiles[0].count, or watch autoscaler decisions in the cluster events.

How to verify: after the new node is Ready, the Pending pod flips to Running within seconds, kubectl get pods -l app=web -o wide shows which node each pod landed on. If pods stay Pending and no node appears, check whether you already hit --max-count 5; the autoscaler cannot exceed it.

Expected output:

$ kubectl get nodes
NAME                                  STATUS   ROLES
aks-nodepool1-12345678-vmss000000      Ready    agent
aks-nodepool1-12345678-vmss000001      Ready    agent
aks-nodepool1-12345678-vmss000002      Ready    agent

What this teaches: The cluster autoscaler reacts to pending pods, not to CPU directly; unschedulable pods are its trigger.