Build Optimized Docker Images with Multi-Stage Dockerfiles
You will write a Dockerfile, build it with multi-stage layers, and scan the result for vulnerabilities. By the end you will produce a small, cache-friendly image with a minimal attack surface.
Most first-pass Dockerfiles work but ship far too much: a full compiler toolchain, package managers, shells, and hundreds of megabytes of libraries the application never calls at runtime. Every extra package is extra attack surface and another stream of CVEs to triage. In this lab you will take a small Go service from a naive ~900MB single-stage image down to a ~12MB distroless runtime, set the build up so rebuilds take seconds instead of minutes thanks to BuildKit layer caching, and finish by scanning the image with Trivy and gating on CRITICAL findings, the same workflow you would run in a CI pipeline before pushing to a registry.
The lab assumes a small HTTP service written in Go (any single-binary language works, the same pattern applies to Rust, Java, or Node with minor changes). All example output is illustrative but representative of Docker 24+ with BuildKit and Trivy 0.5x.
Learning Objectives
- Author a Dockerfile using layer-cache-friendly ordering.
- Shrink images with a multi-stage build.
- Scan images for known CVEs with Trivy.
- Use a
.dockerignorefile to keep secrets and clutter out of the build context. - Inspect image layers with
docker historyand explain where image size comes from. - Time: ~3 hours · Difficulty: Intermediate · Prereqs: Docker and a small app to containerize.
Architecture Overview
A multi-stage Dockerfile contains two (or more) FROM instructions. The first stage is a build environment: it starts from a fat base image like golang:1.22 (~800MB) that contains the compiler, module tooling, and headers. The second stage is the runtime environment: it starts from a minimal base, here gcr.io/distroless/static, which contains nothing but CA certificates, timezone data, and a non-root user. There is no shell, no package manager, and no coreutils, which is precisely the point from a security perspective.
The only thing that crosses the boundary between stages is what you explicitly copy with COPY --from=build. The Go compiler produces a statically linked binary (because CGO_ENABLED=0 removes the dependency on libc), so the runtime stage needs literally nothing else. The final image is the only stage that gets tagged and pushed; the build stage is discarded after the build, though BuildKit keeps its layers in the local cache to speed up the next run.
compile/deps] B -->|copy artifact| R[Runtime stage
slim base] R --> Img[Final image] Img -->|Trivy scan| Scan[CVE report]
Two ideas in this diagram do most of the heavy lifting:
- Layer caching: every instruction in a Dockerfile produces a layer. If the inputs to an instruction have not changed since the last build, BuildKit reuses the cached layer instead of re-running it. Ordering the Dockerfile from least-frequently-changing to most-frequently-changing (dependencies before source code) is what makes rebuilds fast.
- Stage separation: vulnerability scanners report on what is in the shipped image. A distroless runtime has almost no packages, so the scan surface collapses, typically from hundreds of findings in a full
golangbase down to zero or a handful.
Environment Setup
Prerequisites
- Docker Engine 24+ with BuildKit enabled (default in recent versions).
- A small sample application with source code to containerize (any language).
- Assumed skills: Docker fundamentals (images vs containers) and basic shell use.
You will need: Docker with BuildKit and Trivy (install docs).
Before you begin: have a buildable app (a small Go or Node service works well).
First confirm Docker is running and that BuildKit, the modern build backend, is available. Docker 23+ enables BuildKit by default, but it is worth checking explicitly because several steps in this lab depend on it:
Verify Docker and BuildKit
docker version
docker buildx version
Install Trivy if you do not already have it. On macOS use Homebrew; on Debian/Ubuntu use the Aqua Security apt repository:
Install Trivy (macOS and Debian/Ubuntu)
# macOS
brew install trivy
# Debian/Ubuntu
sudo apt-get install wget gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update && sudo apt-get install trivy
Verify Trivy
trivy version
For the sample app, a minimal Go module is enough. The exact code does not matter, what matters is that it compiles to a static binary. A typical project layout for this lab:
Project layout used in this lab
myapp/
main.go # a tiny HTTP server listening on :8080
go.mod # module example.com/myapp; go 1.22
go.sum # dependency checksums (may be empty of third-party deps)
Dockerfile
.dockerignore
Common mistake: skipping the go.mod file. Since Go 1.16 the toolchain builds in module mode by default, and go build . inside a directory with no go.mod fails with go: go.mod file not found. Run go mod init example.com/myapp locally first if your project predates modules.
Step-by-Step Execution
Compiling in a build stage and copying only the artifact into a slim base keeps the runtime tiny.
# Dockerfile
FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o app .
FROM gcr.io/distroless/static
COPY --from=build /src/app /app
ENTRYPOINT ["/app"]
Expected output:
$ docker build --check -f Dockerfile . && echo "Dockerfile lints clean"
Dockerfile lints clean
What each line does and why:
FROM golang:1.22 AS build, starts the build stage from the official Go image and names itbuildso later instructions can reference it. Pinning the minor version is steadier thanlatest, but the tag still floats with patch releases, only a digest (golang:1.22@sha256:...) makes a build truly reproducible.WORKDIR /src, sets the working directory for subsequent instructions; avoids scattering files across the image root.CGO_ENABLED=0, forces a fully static binary with no libc dependency. This is what allows the binary to run ondistroless/static, which has no C library at all.ENTRYPOINT ["/app"], exec form (JSON array), so the binary runs as PID 1 and receives signals like SIGTERM directly. The shell form (ENTRYPOINT /app) wraps the process insh -c, which would fail here anyway because distroless has no shell.
The minimal Dockerfile above works, but it wastes the layer cache: COPY . . copies the entire source tree before the build, so any file change, even editing a README, invalidates every layer below it and forces a full re-download of dependencies. The cache-friendly version copies the module files first and downloads dependencies as a separate layer:
Dockerfile, cache-optimized version
# syntax=docker/dockerfile:1
FROM golang:1.22 AS build
WORKDIR /src
# 1. Dependencies change rarely - cache them in their own layer
COPY go.mod go.sum ./
RUN go mod download
# 2. Source code changes often - copy it last
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o app .
FROM gcr.io/distroless/static:nonroot
COPY --from=build /src/app /app
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app"]
With this ordering, editing main.go only invalidates the layers from COPY . . downward, the go mod download layer is reused and rebuilds drop from minutes to seconds. The extra flags matter too: -trimpath strips local filesystem paths from the binary (reproducibility and information hygiene), and -ldflags="-s -w" omits the symbol and debug tables, shaving roughly 20-30% off the binary size. The runtime stage uses the nonroot tag and USER nonroot:nonroot so the process never runs as root, a cheap, meaningful hardening step.
.dockerignore, keep the build context clean
.git
.gitignore
Dockerfile
.dockerignore
*.md
.env
.env.*
bin/
tmp/
node_modules/
The .dockerignore file controls what gets sent to the Docker daemon as the build context. Without it, COPY . . can silently bake your .git directory, including full history and any secrets ever committed, into an image layer, where it stays even if you delete it in a later layer (earlier layers are immutable and remain in the image). Excluding .env files is the same story: dotenv files are a classic source of leaked credentials in published images.
How to verify: run docker build twice (next step) and watch for CACHED markers beside the go mod download step on the second run. To confirm the context is clean, check the build output line transferring context, it should be kilobytes, not tens of megabytes.
What this teaches: Multi-stage Dockerfiles separate the build toolchain from the runtime; only the artifact ships.
DOCKER_BUILDKIT=1 docker build -t tyf/app:1.0 .Breaking the command down: DOCKER_BUILDKIT=1 forces the BuildKit backend (redundant on Docker 23+, but harmless and explicit), -t tyf/app:1.0 names and tags the image as tyf/app version 1.0, and the trailing . is the build context, the directory sent to the daemon, filtered through .dockerignore.
On the first run BuildKit pulls both base images and executes every instruction. Use --progress=plain if you want full command output instead of the interactive display, essential in CI logs:
First build (truncated output)
DOCKER_BUILDKIT=1 docker build --progress=plain -t tyf/app:1.0 .
Expected output:
$ docker build -t demo:multi .
[+] Building 12.4s (14/14) FINISHED
=> naming to docker.io/library/demo:multi
Now change one line in main.go and build again. Every step up to COPY . . should report CACHED, and only the compile and export steps re-run:
Tagging discipline matters. tyf/app:1.0 follows the pattern registry-namespace/name:tag. Avoid shipping only latest: mutable tags make rollbacks and incident forensics painful because you cannot tell which build is actually running. A common convention is to tag each build with both a semantic version and the git commit SHA.
How to verify: docker image inspect tyf/app:1.0 --format '{{.Os}}/{{.Architecture}} {{json .Config.Entrypoint}}' should print linux/amd64 ["/app"]. Common mistake: forgetting the trailing . context, which makes Docker read the Dockerfile from stdin and produce a confusing the Dockerfile cannot be empty error.
What this teaches: Layer caching rewards good instruction ordering; put slow, stable steps early and fast, changing steps late.
Numbers make the multi-stage win concrete. First build a naive single-stage version for comparison, the kind of Dockerfile most people write on day one, which ships the entire Go toolchain as the runtime:
Dockerfile.naive, single-stage, for comparison only
FROM golang:1.22
WORKDIR /src
COPY . .
RUN go build -o app .
ENTRYPOINT ["/app"]
Expected output:
$ docker images demo --format '{{.Tag}} {{.Size}}'
single 342MB
multi 48MB
Build both and compare
docker build -f Dockerfile.naive -t tyf/app:naive .
docker images tyf/app
That is roughly a 98% reduction: 892MB down to 12.3MB. Smaller images pull faster, start faster, cost less to store in a registry, and, the security-relevant part, contain drastically fewer packages that can carry CVEs.
To understand where the remaining size lives, inspect the layer history. Each instruction that modified the filesystem contributed a layer:
Inspect image layers
docker history tyf/app:1.0
Reading this bottom-up: the distroless base contributes ~2.2MB (CA certs, tzdata, passwd entries), and the only meaningful addition is the compiled binary at ~10.1MB. Metadata-only instructions like ENTRYPOINT, USER, and EXPOSE cost 0 bytes.
Common mistake: trying to shrink an image by deleting files in a later RUN, e.g. RUN apt-get install ... followed by RUN rm -rf /var/lib/apt/lists in a separate instruction. Layers are additive and immutable: the files still exist in the earlier layer and still count toward the pull size. Cleanup must happen in the same RUN as the install (RUN apt-get update && apt-get install -y foo && rm -rf /var/lib/apt/lists/*), or the data must never enter the image, which is exactly what multi-stage builds achieve for toolchains.
How to verify: confirm the final stage carries no shell or package manager by trying to run one: docker run --rm --entrypoint sh tyf/app:1.0 should fail with an error like exec: "sh": not found. That failure is a feature, an attacker who compromises the process gets no shell and no tooling.
What this teaches: Size comparison makes bloat visible; you cannot optimize what you do not measure.
trivy image --severity HIGH,CRITICAL tyf/app:1.0Trivy inspects the image's package inventory (OS packages plus language dependencies such as Go modules) and matches each component against its vulnerability database. On the very first run it downloads that database (~50MB) to ~/.cache/trivy, so the first scan needs internet access; subsequent runs reuse the cached DB and refresh it periodically.
Scanning the naive single-stage image shows why base image choice dominates your security posture:
The same scan against the multi-stage image tells a very different story:
Distroless images ship almost no packages, so there is almost nothing to be vulnerable. Any findings that do appear against a distroless base are usually in the Go standard library compiled into your binary, and the fix is to rebuild with a patched Go version, which is why pinning and regularly bumping FROM golang:1.22 matters.
To turn the scan into a gate, use --exit-code 1 so Trivy returns non-zero when it finds anything at or above your threshold, the exact behaviour you want blocking a CI pipeline:
Fail the build on CRITICAL findings
trivy image --exit-code 1 --severity CRITICAL tyf/app:1.0
echo "exit code: $?"
Expected output:
$ trivy image --exit-code 1 --severity CRITICAL tyf/app:1.0
tyf/app:1.0 (distroless static)
Total: 0 (CRITICAL: 0)
$ echo "exit code: $?"
exit code: 0
If you need to accept a specific finding temporarily (for example a CVE with no fixed version that you have assessed as not exploitable in your deployment), list it, with a justification comment, in a .trivyignore file at the repo root rather than lowering the severity threshold globally. Ignored CVEs stay visible in version control and reviewable.
Common mistake: scanning the wrong artifact. Scan the final image you actually ship, not the build-stage image or the local Dockerfile. trivy config Dockerfile (misconfiguration scanning) and trivy fs . (dependency scanning) are useful complements, but neither replaces scanning the built image.
How to verify: deliberately scan the naive image and confirm Trivy exits 1 with --exit-code 1, you now know the gate actually fails when it should, not just passes when it should.
What this teaches: Scanning before shipping turns known vulnerabilities into a decision instead of a surprise.
Step Flow Map
Each box is one step in the lab; work them in order, since later steps build on earlier state.
Progress So Far
How It Works
Each Dockerfile instruction creates a layer, and layers cache: if the instruction and its inputs are unchanged, the build reuses the cached layer instead of re-executing. Ordering the Dockerfile from least-frequently-changed (base image, dependency install) to most-frequently-changed (your source copy) is what makes rebuilds take seconds instead of minutes. A change early in the file invalidates every layer after it.
Multi-stage builds separate the build environment from the runtime environment. The first stage carries compilers and SDKs to produce an artifact; the final stage copies only that artifact into a minimal base. The result is an image that contains your application but not your toolchain, which is smaller, faster to pull, and presents less attack surface.
Scanners work by comparing the packages in your image's layers against vulnerability databases. They report CVEs with severities, but the fix is almost always the same: rebuild from a newer base image or bump the vulnerable package. Scanning in the build pipeline turns shipping known CVEs from a discovery into a choice.
Testing & Validation
Before calling the lab done, prove the image actually works, not just that it builds and scans clean. Run the container and exercise it the way a user would:
Smoke-test the running container
docker run --rm -d --name app-test -p 8080:8080 tyf/app:1.0
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/
docker logs app-test
docker stop app-test
docker image inspect tyf/app:1.0 --format 'user={{.Config.User}} arch={{.Architecture}}' && trivy image --exit-code 1 --severity CRITICAL tyf/app:1.0The app should run and Trivy should exit 0 (no criticals). If so, you have a small, scanned image.
Two things worth knowing when validating distroless images specifically:
- You cannot
docker exec -it app-test sh, there is no shell to exec into. For debugging, distroless publishes a:debugvariant (e.g.gcr.io/distroless/static:debug-nonroot) that bundles busybox; build a debug tag of your image from it when you need a shell, and keep it out of production. - Check the process identity:
docker top app-testshould show the process running as thenonrootUID (65532), confirming theUSERdirective took effect.
Troubleshooting
Issues that come up most often in this lab, with the cause and the fix:
- Image still huge. The final stage almost certainly is not the slim one. Confirm the last
FROMin the Dockerfile is the distroless/alpine base, that you copy the artifact withCOPY --from=build(the documented double-dash form), and that you tagged the result of the whole build. Rundocker history tyf/app:1.0and look for fat layers carrying the toolchain, if you see them, you are shipping the build stage. - Cache never hits; every build re-downloads dependencies. The Dockerfile copies source before resolving dependencies. Reorder so
COPY go.mod go.sum ./andRUN go mod downloadcome beforeCOPY . ., and add a.dockerignoreso files that change constantly (.git, logs, editor temp files) do not invalidate the context. Also confirm BuildKit is actually on: legacy builders (DOCKER_BUILDKIT=0) cache far less aggressively. - Trivy DB error on first run (
failed to download vulnerability DB). Trivy needs internet access once to fetch its database. Behind a corporate proxy, exportHTTPS_PROXYbefore scanning. In fully offline environments, pre-seed the cache directory (~/.cache/trivy) from a machine that has access, or scan with--skip-db-updateagainst an older cached DB and record the DB age in your report. - Build fails with
go: go.mod file not found. The project has no module file but Go 1.22 builds in module mode. Rungo mod init example.com/myapplocally, commit the resultinggo.mod(andgo.sum), and rebuild. As a stopgap for trivial single-file programs only,GO111MODULE=off go buildworks, but modules are the correct fix. - Container exits instantly with
exec /app: no such file or directory. The binary exists but its interpreter does not: it was dynamically linked (CGO enabled) and the distroless base has no libc. Rebuild withCGO_ENABLED=0, or switch the runtime base togcr.io/distroless/base-nossl(orbase) if the app genuinely needs cgo. A related variant,exec format error, means an architecture mismatch, e.g. building on an Apple Silicon Mac (arm64) and deploying to an amd64 server; fix withdocker buildx build --platform linux/amd64 -t tyf/app:1.0 ..
Quick-reference for the failures you are most likely to hit in this lab.
| Symptom | Likely cause | Fix |
|---|---|---|
| Builds are slow every time | A frequently changing instruction sits early in the Dockerfile, invalidating the cache | Move COPY . . after dependency installation so code changes do not reinstall dependencies. |
COPY failed: file not found | The file is outside the build context or excluded by .dockerignore | Check the context directory and the ignore file; the daemon only sees what you send it. |
| Scanner reports critical CVEs | The base image or a package has known vulnerabilities | Bump the base tag, rebuild, and re-scan; pin the new digest once it is clean. |
| Final image is still huge | Build artifacts or caches leaked into the final stage | Verify only the artifact is copied with --from, and add a .dockerignore. |
| Tag confusion across environments | latest moved between deploy and rollback | Tag releases immutably (for example with the git SHA) and pin digests in production. |
Extension Ideas
- Push to a registry and sign the image with cosign.
- Generate an SBOM (
trivy image --format spdx-json -o sbom.json tyf/app:1.0) and archive it alongside each release. - Add the scan to a pipeline in Full CI/CD Pipeline.
- Run it on a cluster in Pods & Deployments.
- Try
docker buildx build --cache-to type=registryto share the build cache across CI runners.
Key Results
- Produced a ~12MB runtime image via multi-stage build.
- Cut rebuild time with BuildKit layer caching.
- Scanned for HIGH/CRITICAL CVEs before shipping.
- Minimized attack surface with a distroless base.
Key Takeaways
- Order your Dockerfile for the cache: dependencies before source code, least-frequently-changing layers first. This is the difference between 6-second and 6-minute rebuilds.
- Ship artifacts, not toolchains: the build stage exists to produce the binary; only the binary crosses into the runtime stage.
- Smaller image, smaller attack surface: distroless removed ~880MB of packages and took the CVE count from triple digits to zero.
- Secrets do not belong in layers: use
.dockerignorefor.gitand.envfiles, deleted files still live in earlier layers. - Scan the image you ship and gate on it:
trivy image --exit-code 1 --severity CRITICALturns scanning from a report into an enforcement point. - Verify behaviour, not just builds: a clean scan does not prove the app runs, smoke-test the container end to end.
Practice Challenges
Challenge 1: Get under 50 MB
Goal: Rebuild the sample app image under 50 MB using a minimal base and multi-stage build.
Hint: Distroless or alpine bases plus copying only the compiled artifact is the standard recipe.
Challenge 2: Drop root
Goal: Add a non-root USER to the final stage and prove the container no longer runs as root.
Hint: Run docker exec app whoami for the proof; fix file ownership with --chown on COPY.
Challenge 3: Compare three strategies
Goal: Build the same app as single-stage, multi-stage, and distroless, and tabulate size and CVE count.
Hint: The table is the deliverable; it makes the tradeoffs visible instead of theoretical.