You will write a Dockerfile, build it with multi-stage layers, and scan the result for vulnerabilities. By the end you will produce a small, cache-friendly image with a minimal attack surface.

Most first-pass Dockerfiles work but ship far too much: a full compiler toolchain, package managers, shells, and hundreds of megabytes of libraries the application never calls at runtime. Every extra package is extra attack surface and another stream of CVEs to triage. In this lab you will take a small Go service from a naive ~900MB single-stage image down to a ~12MB distroless runtime, set the build up so rebuilds take seconds instead of minutes thanks to BuildKit layer caching, and finish by scanning the image with Trivy and gating on CRITICAL findings, the same workflow you would run in a CI pipeline before pushing to a registry.

The lab assumes a small HTTP service written in Go (any single-binary language works, the same pattern applies to Rust, Java, or Node with minor changes). All example output is illustrative but representative of Docker 24+ with BuildKit and Trivy 0.5x.

Learning Objectives

  • Author a Dockerfile using layer-cache-friendly ordering.
  • Shrink images with a multi-stage build.
  • Scan images for known CVEs with Trivy.
  • Use a .dockerignore file to keep secrets and clutter out of the build context.
  • Inspect image layers with docker history and explain where image size comes from.
  • Time: ~3 hours · Difficulty: Intermediate · Prereqs: Docker and a small app to containerize.

Architecture Overview

A multi-stage Dockerfile contains two (or more) FROM instructions. The first stage is a build environment: it starts from a fat base image like golang:1.22 (~800MB) that contains the compiler, module tooling, and headers. The second stage is the runtime environment: it starts from a minimal base, here gcr.io/distroless/static, which contains nothing but CA certificates, timezone data, and a non-root user. There is no shell, no package manager, and no coreutils, which is precisely the point from a security perspective.

The only thing that crosses the boundary between stages is what you explicitly copy with COPY --from=build. The Go compiler produces a statically linked binary (because CGO_ENABLED=0 removes the dependency on libc), so the runtime stage needs literally nothing else. The final image is the only stage that gets tagged and pushed; the build stage is discarded after the build, though BuildKit keeps its layers in the local cache to speed up the next run.

Two ideas in this diagram do most of the heavy lifting:

  • Layer caching: every instruction in a Dockerfile produces a layer. If the inputs to an instruction have not changed since the last build, BuildKit reuses the cached layer instead of re-running it. Ordering the Dockerfile from least-frequently-changing to most-frequently-changing (dependencies before source code) is what makes rebuilds fast.
  • Stage separation: vulnerability scanners report on what is in the shipped image. A distroless runtime has almost no packages, so the scan surface collapses, typically from hundreds of findings in a full golang base down to zero or a handful.

Environment Setup

Prerequisites

  • Docker Engine 24+ with BuildKit enabled (default in recent versions).
  • A small sample application with source code to containerize (any language).
  • Assumed skills: Docker fundamentals (images vs containers) and basic shell use.

You will need: Docker with BuildKit and Trivy (install docs).

Before you begin: have a buildable app (a small Go or Node service works well).

First confirm Docker is running and that BuildKit, the modern build backend, is available. Docker 23+ enables BuildKit by default, but it is worth checking explicitly because several steps in this lab depend on it:

Verify Docker and BuildKit

docker version
docker buildx version
$ docker version --format 'Client: {{.Client.Version}} Server: {{.Server.Version}}'
Client: 24.0.7 Server: 24.0.7
$ docker buildx version
github.com/docker/buildx v0.12.0

Install Trivy if you do not already have it. On macOS use Homebrew; on Debian/Ubuntu use the Aqua Security apt repository:

Install Trivy (macOS and Debian/Ubuntu)

# macOS
brew install trivy

# Debian/Ubuntu
sudo apt-get install wget gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update && sudo apt-get install trivy

Verify Trivy

trivy version
$ trivy version
Version: 0.58.1

For the sample app, a minimal Go module is enough. The exact code does not matter, what matters is that it compiles to a static binary. A typical project layout for this lab:

Project layout used in this lab

myapp/
  main.go        # a tiny HTTP server listening on :8080
  go.mod         # module example.com/myapp; go 1.22
  go.sum         # dependency checksums (may be empty of third-party deps)
  Dockerfile
  .dockerignore

Common mistake: skipping the go.mod file. Since Go 1.16 the toolchain builds in module mode by default, and go build . inside a directory with no go.mod fails with go: go.mod file not found. Run go mod init example.com/myapp locally first if your project predates modules.

Step-by-Step Execution

01
Write a multi-stage Dockerfile

Compiling in a build stage and copying only the artifact into a slim base keeps the runtime tiny.

# Dockerfile
FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o app .

FROM gcr.io/distroless/static
COPY --from=build /src/app /app
ENTRYPOINT ["/app"]

Expected output:

$ docker build --check -f Dockerfile . && echo "Dockerfile lints clean"
Dockerfile lints clean

What each line does and why:

  • FROM golang:1.22 AS build, starts the build stage from the official Go image and names it build so later instructions can reference it. Pinning the minor version is steadier than latest, but the tag still floats with patch releases, only a digest (golang:1.22@sha256:...) makes a build truly reproducible.
  • WORKDIR /src, sets the working directory for subsequent instructions; avoids scattering files across the image root.
  • CGO_ENABLED=0, forces a fully static binary with no libc dependency. This is what allows the binary to run on distroless/static, which has no C library at all.
  • ENTRYPOINT ["/app"], exec form (JSON array), so the binary runs as PID 1 and receives signals like SIGTERM directly. The shell form (ENTRYPOINT /app) wraps the process in sh -c, which would fail here anyway because distroless has no shell.

The minimal Dockerfile above works, but it wastes the layer cache: COPY . . copies the entire source tree before the build, so any file change, even editing a README, invalidates every layer below it and forces a full re-download of dependencies. The cache-friendly version copies the module files first and downloads dependencies as a separate layer:

Dockerfile, cache-optimized version

# syntax=docker/dockerfile:1
FROM golang:1.22 AS build
WORKDIR /src

# 1. Dependencies change rarely - cache them in their own layer
COPY go.mod go.sum ./
RUN go mod download

# 2. Source code changes often - copy it last
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o app .

FROM gcr.io/distroless/static:nonroot
COPY --from=build /src/app /app
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app"]

With this ordering, editing main.go only invalidates the layers from COPY . . downward, the go mod download layer is reused and rebuilds drop from minutes to seconds. The extra flags matter too: -trimpath strips local filesystem paths from the binary (reproducibility and information hygiene), and -ldflags="-s -w" omits the symbol and debug tables, shaving roughly 20-30% off the binary size. The runtime stage uses the nonroot tag and USER nonroot:nonroot so the process never runs as root, a cheap, meaningful hardening step.

.dockerignore, keep the build context clean

.git
.gitignore
Dockerfile
.dockerignore
*.md
.env
.env.*
bin/
tmp/
node_modules/

The .dockerignore file controls what gets sent to the Docker daemon as the build context. Without it, COPY . . can silently bake your .git directory, including full history and any secrets ever committed, into an image layer, where it stays even if you delete it in a later layer (earlier layers are immutable and remain in the image). Excluding .env files is the same story: dotenv files are a classic source of leaked credentials in published images.

How to verify: run docker build twice (next step) and watch for CACHED markers beside the go mod download step on the second run. To confirm the context is clean, check the build output line transferring context, it should be kilobytes, not tens of megabytes.

What this teaches: Multi-stage Dockerfiles separate the build toolchain from the runtime; only the artifact ships.

02
Build with cache and a tag
DOCKER_BUILDKIT=1 docker build -t tyf/app:1.0 .
BuildKit caches layers, speeding up rebuilds.

Breaking the command down: DOCKER_BUILDKIT=1 forces the BuildKit backend (redundant on Docker 23+, but harmless and explicit), -t tyf/app:1.0 names and tags the image as tyf/app version 1.0, and the trailing . is the build context, the directory sent to the daemon, filtered through .dockerignore.

On the first run BuildKit pulls both base images and executes every instruction. Use --progress=plain if you want full command output instead of the interactive display, essential in CI logs:

First build (truncated output)

DOCKER_BUILDKIT=1 docker build --progress=plain -t tyf/app:1.0 .

Expected output:

$ docker build -t demo:multi .
[+] Building 12.4s (14/14) FINISHED
 => naming to docker.io/library/demo:multi
$ DOCKER_BUILDKIT=1 docker build -t tyf/app:1.0 .
[+] Building 47.3s (12/12) FINISHED => [internal] load build definition from Dockerfile => [internal] load .dockerignore => [internal] load metadata for docker.io/library/golang:1.22 => [internal] load metadata for gcr.io/distroless/static:nonroot => [build 1/5] FROM docker.io/library/golang:1.22 => [internal] load build context => [build 2/5] WORKDIR /src => [build 3/5] COPY go.mod go.sum ./ => [build 4/5] RUN go mod download => [build 5/5] COPY . . => [build 6/5] RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o app . => [stage-1 2/3] COPY --from=build /src/app /app => exporting to image => => writing image sha256:9f3c...a21b => => naming to docker.io/tyf/app:1.0

Now change one line in main.go and build again. Every step up to COPY . . should report CACHED, and only the compile and export steps re-run:

$ DOCKER_BUILDKIT=1 docker build -t tyf/app:1.1 .
[+] Building 6.1s (12/12) FINISHED => [build 3/5] COPY go.mod go.sum ./ CACHED => [build 4/5] RUN go mod download CACHED => [build 5/5] COPY . . => [build 6/5] RUN CGO_ENABLED=0 GOOS=linux go build ... => exporting to image => => naming to docker.io/tyf/app:1.1

Tagging discipline matters. tyf/app:1.0 follows the pattern registry-namespace/name:tag. Avoid shipping only latest: mutable tags make rollbacks and incident forensics painful because you cannot tell which build is actually running. A common convention is to tag each build with both a semantic version and the git commit SHA.

How to verify: docker image inspect tyf/app:1.0 --format '{{.Os}}/{{.Architecture}} {{json .Config.Entrypoint}}' should print linux/amd64 ["/app"]. Common mistake: forgetting the trailing . context, which makes Docker read the Dockerfile from stdin and produce a confusing the Dockerfile cannot be empty error.

What this teaches: Layer caching rewards good instruction ordering; put slow, stable steps early and fast, changing steps late.

03
Compare image size

Numbers make the multi-stage win concrete. First build a naive single-stage version for comparison, the kind of Dockerfile most people write on day one, which ships the entire Go toolchain as the runtime:

Dockerfile.naive, single-stage, for comparison only

FROM golang:1.22
WORKDIR /src
COPY . .
RUN go build -o app .
ENTRYPOINT ["/app"]

Expected output:

$ docker images demo --format '{{.Tag}} {{.Size}}'
single  342MB
multi   48MB

Build both and compare

docker build -f Dockerfile.naive -t tyf/app:naive .
docker images tyf/app
$ docker images tyf/app
REPOSITORY TAG SIZE tyf/app 1.0 12.3MB <- distroless runtime tyf/app naive 892MB <- single-stage golang base

That is roughly a 98% reduction: 892MB down to 12.3MB. Smaller images pull faster, start faster, cost less to store in a registry, and, the security-relevant part, contain drastically fewer packages that can carry CVEs.

To understand where the remaining size lives, inspect the layer history. Each instruction that modified the filesystem contributed a layer:

Inspect image layers

docker history tyf/app:1.0
$ docker history tyf/app:1.0
IMAGE CREATED CREATED BY SIZE 9f3ca21b 2 minutes ago ENTRYPOINT ["/app"] 0B <missing> 2 minutes ago EXPOSE map[8080/tcp:{}] 0B <missing> 2 minutes ago USER nonroot:nonroot 0B <missing> 2 minutes ago COPY /src/app /app # buildkit 10.1MB <missing> 5 weeks ago COPY / / # distroless static files 2.2MB

Reading this bottom-up: the distroless base contributes ~2.2MB (CA certs, tzdata, passwd entries), and the only meaningful addition is the compiled binary at ~10.1MB. Metadata-only instructions like ENTRYPOINT, USER, and EXPOSE cost 0 bytes.

Common mistake: trying to shrink an image by deleting files in a later RUN, e.g. RUN apt-get install ... followed by RUN rm -rf /var/lib/apt/lists in a separate instruction. Layers are additive and immutable: the files still exist in the earlier layer and still count toward the pull size. Cleanup must happen in the same RUN as the install (RUN apt-get update && apt-get install -y foo && rm -rf /var/lib/apt/lists/*), or the data must never enter the image, which is exactly what multi-stage builds achieve for toolchains.

How to verify: confirm the final stage carries no shell or package manager by trying to run one: docker run --rm --entrypoint sh tyf/app:1.0 should fail with an error like exec: "sh": not found. That failure is a feature, an attacker who compromises the process gets no shell and no tooling.

What this teaches: Size comparison makes bloat visible; you cannot optimize what you do not measure.

04
Scan for vulnerabilities
trivy image --severity HIGH,CRITICAL tyf/app:1.0

Trivy inspects the image's package inventory (OS packages plus language dependencies such as Go modules) and matches each component against its vulnerability database. On the very first run it downloads that database (~50MB) to ~/.cache/trivy, so the first scan needs internet access; subsequent runs reuse the cached DB and refresh it periodically.

Scanning the naive single-stage image shows why base image choice dominates your security posture:

$ trivy image --severity HIGH,CRITICAL tyf/app:naive
tyf/app:naive (debian 12.5) Total: 147 (HIGH: 118, CRITICAL: 29) Library Vulnerability Severity Installed Fixed Title stdlib CVE-2024-24790 CRITICAL 1.22.0 1.22.4 net/netip: unexpected behavior... openssl CVE-2023-5678 HIGH 3.0.11 3.0.12 Excessive time spent checking DH keys...

The same scan against the multi-stage image tells a very different story:

$ trivy image --severity HIGH,CRITICAL tyf/app:1.0
tyf/app:1.0 (distroless static) Total: 0 (HIGH: 0, CRITICAL: 0)

Distroless images ship almost no packages, so there is almost nothing to be vulnerable. Any findings that do appear against a distroless base are usually in the Go standard library compiled into your binary, and the fix is to rebuild with a patched Go version, which is why pinning and regularly bumping FROM golang:1.22 matters.

To turn the scan into a gate, use --exit-code 1 so Trivy returns non-zero when it finds anything at or above your threshold, the exact behaviour you want blocking a CI pipeline:

Fail the build on CRITICAL findings

trivy image --exit-code 1 --severity CRITICAL tyf/app:1.0
echo "exit code: $?"

Expected output:

$ trivy image --exit-code 1 --severity CRITICAL tyf/app:1.0
tyf/app:1.0 (distroless static)
Total: 0 (CRITICAL: 0)
$ echo "exit code: $?"
exit code: 0
$ echo "exit code: $?"
exit code: 0 <- no CRITICAL findings, gate passes

If you need to accept a specific finding temporarily (for example a CVE with no fixed version that you have assessed as not exploitable in your deployment), list it, with a justification comment, in a .trivyignore file at the repo root rather than lowering the severity threshold globally. Ignored CVEs stay visible in version control and reviewable.

Common mistake: scanning the wrong artifact. Scan the final image you actually ship, not the build-stage image or the local Dockerfile. trivy config Dockerfile (misconfiguration scanning) and trivy fs . (dependency scanning) are useful complements, but neither replaces scanning the built image.

How to verify: deliberately scan the naive image and confirm Trivy exits 1 with --exit-code 1, you now know the gate actually fails when it should, not just passes when it should.

What this teaches: Scanning before shipping turns known vulnerabilities into a decision instead of a surprise.

Step Flow Map

Each box is one step in the lab; work them in order, since later steps build on earlier state.

Progress So Far

How It Works

Each Dockerfile instruction creates a layer, and layers cache: if the instruction and its inputs are unchanged, the build reuses the cached layer instead of re-executing. Ordering the Dockerfile from least-frequently-changed (base image, dependency install) to most-frequently-changed (your source copy) is what makes rebuilds take seconds instead of minutes. A change early in the file invalidates every layer after it.

Multi-stage builds separate the build environment from the runtime environment. The first stage carries compilers and SDKs to produce an artifact; the final stage copies only that artifact into a minimal base. The result is an image that contains your application but not your toolchain, which is smaller, faster to pull, and presents less attack surface.

Scanners work by comparing the packages in your image's layers against vulnerability databases. They report CVEs with severities, but the fix is almost always the same: rebuild from a newer base image or bump the vulnerable package. Scanning in the build pipeline turns shipping known CVEs from a discovery into a choice.

Testing & Validation

Before calling the lab done, prove the image actually works, not just that it builds and scans clean. Run the container and exercise it the way a user would:

Smoke-test the running container

docker run --rm -d --name app-test -p 8080:8080 tyf/app:1.0
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/
docker logs app-test
docker stop app-test
$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/
200
$ docker logs app-test
2026/09/21 14:03:11 listening on :8080
docker image inspect tyf/app:1.0 --format 'user={{.Config.User}} arch={{.Architecture}}' && trivy image --exit-code 1 --severity CRITICAL tyf/app:1.0

The app should run and Trivy should exit 0 (no criticals). If so, you have a small, scanned image.

Two things worth knowing when validating distroless images specifically:

  • You cannot docker exec -it app-test sh, there is no shell to exec into. For debugging, distroless publishes a :debug variant (e.g. gcr.io/distroless/static:debug-nonroot) that bundles busybox; build a debug tag of your image from it when you need a shell, and keep it out of production.
  • Check the process identity: docker top app-test should show the process running as the nonroot UID (65532), confirming the USER directive took effect.

Troubleshooting

Issues that come up most often in this lab, with the cause and the fix:

  • Image still huge. The final stage almost certainly is not the slim one. Confirm the last FROM in the Dockerfile is the distroless/alpine base, that you copy the artifact with COPY --from=build (the documented double-dash form), and that you tagged the result of the whole build. Run docker history tyf/app:1.0 and look for fat layers carrying the toolchain, if you see them, you are shipping the build stage.
  • Cache never hits; every build re-downloads dependencies. The Dockerfile copies source before resolving dependencies. Reorder so COPY go.mod go.sum ./ and RUN go mod download come before COPY . ., and add a .dockerignore so files that change constantly (.git, logs, editor temp files) do not invalidate the context. Also confirm BuildKit is actually on: legacy builders (DOCKER_BUILDKIT=0) cache far less aggressively.
  • Trivy DB error on first run (failed to download vulnerability DB). Trivy needs internet access once to fetch its database. Behind a corporate proxy, export HTTPS_PROXY before scanning. In fully offline environments, pre-seed the cache directory (~/.cache/trivy) from a machine that has access, or scan with --skip-db-update against an older cached DB and record the DB age in your report.
  • Build fails with go: go.mod file not found. The project has no module file but Go 1.22 builds in module mode. Run go mod init example.com/myapp locally, commit the resulting go.mod (and go.sum), and rebuild. As a stopgap for trivial single-file programs only, GO111MODULE=off go build works, but modules are the correct fix.
  • Container exits instantly with exec /app: no such file or directory. The binary exists but its interpreter does not: it was dynamically linked (CGO enabled) and the distroless base has no libc. Rebuild with CGO_ENABLED=0, or switch the runtime base to gcr.io/distroless/base-nossl (or base) if the app genuinely needs cgo. A related variant, exec format error, means an architecture mismatch, e.g. building on an Apple Silicon Mac (arm64) and deploying to an amd64 server; fix with docker buildx build --platform linux/amd64 -t tyf/app:1.0 ..

Quick-reference for the failures you are most likely to hit in this lab.

SymptomLikely causeFix
Builds are slow every timeA frequently changing instruction sits early in the Dockerfile, invalidating the cacheMove COPY . . after dependency installation so code changes do not reinstall dependencies.
COPY failed: file not foundThe file is outside the build context or excluded by .dockerignoreCheck the context directory and the ignore file; the daemon only sees what you send it.
Scanner reports critical CVEsThe base image or a package has known vulnerabilitiesBump the base tag, rebuild, and re-scan; pin the new digest once it is clean.
Final image is still hugeBuild artifacts or caches leaked into the final stageVerify only the artifact is copied with --from, and add a .dockerignore.
Tag confusion across environmentslatest moved between deploy and rollbackTag releases immutably (for example with the git SHA) and pin digests in production.

Extension Ideas

  • Push to a registry and sign the image with cosign.
  • Generate an SBOM (trivy image --format spdx-json -o sbom.json tyf/app:1.0) and archive it alongside each release.
  • Add the scan to a pipeline in Full CI/CD Pipeline.
  • Run it on a cluster in Pods & Deployments.
  • Try docker buildx build --cache-to type=registry to share the build cache across CI runners.

Key Results

  • Produced a ~12MB runtime image via multi-stage build.
  • Cut rebuild time with BuildKit layer caching.
  • Scanned for HIGH/CRITICAL CVEs before shipping.
  • Minimized attack surface with a distroless base.

Key Takeaways

  • Order your Dockerfile for the cache: dependencies before source code, least-frequently-changing layers first. This is the difference between 6-second and 6-minute rebuilds.
  • Ship artifacts, not toolchains: the build stage exists to produce the binary; only the binary crosses into the runtime stage.
  • Smaller image, smaller attack surface: distroless removed ~880MB of packages and took the CVE count from triple digits to zero.
  • Secrets do not belong in layers: use .dockerignore for .git and .env files, deleted files still live in earlier layers.
  • Scan the image you ship and gate on it: trivy image --exit-code 1 --severity CRITICAL turns scanning from a report into an enforcement point.
  • Verify behaviour, not just builds: a clean scan does not prove the app runs, smoke-test the container end to end.

Practice Challenges

Challenge 1: Get under 50 MB

Goal: Rebuild the sample app image under 50 MB using a minimal base and multi-stage build.

Hint: Distroless or alpine bases plus copying only the compiled artifact is the standard recipe.

Challenge 2: Drop root

Goal: Add a non-root USER to the final stage and prove the container no longer runs as root.

Hint: Run docker exec app whoami for the proof; fix file ownership with --chown on COPY.

Challenge 3: Compare three strategies

Goal: Build the same app as single-stage, multi-stage, and distroless, and tabulate size and CVE count.

Hint: The table is the deliverable; it makes the tradeoffs visible instead of theoretical.