Secure Robot-Fleet Updates

Overview

The Army's MINERVA challenges say it plainly: Canada has bought foreign robotic systems of mixed quality, with components from unstable supply chains. The Defence Drone Initiative (launched July 23, 2026) makes supply-chain integrity a theme rather than an afterthought. A drone or ground robot that will accept any firmware image anyone hands it is a liability, no matter how good its sensors are.

This build connects my DevOps discipline to hardware. The fleet is small, the Raspberry Pi nodes from my rover and sensor builds, but the rules are the ones a real program needs:

• Every device boots only signed code, a secure-boot chain from the bootloader up, with a TPM 2.0 measuring each stage and holding the disk keys hostage to that measurement.
• Every update is signed by an offline key, Ed25519 keys that live on YubiKeys, not on any machine connected to a network.
• A bad update rolls itself back, A/B partitions with health checks, so a broken image never strands a node in the field.

And because these robots are meant to operate in the cold, the same cold the MINERVA documents complain about, the whole update-and-rollback cycle gets validated in a freezer, not just on a warm bench.

Proof of Build

The rig on my bench: a Raspberry Pi fleet node wired to its TPM 2.0 module, with the oscilloscope alongside and the laptop showing a verified secure-boot log, TPM integrity status, and the rollback interface after a forced bad-update test.

Bill of Materials

Everything below is already purchased and on my bench. The fleet nodes themselves are shared with the rover and sensor builds, so the incremental cost of making the whole fleet trustworthy is essentially the TPM and the keys.

Architecture

Figure 1: images are signed by a key that never touches a network, delivered through a store-and-forward campaign queue, and installed into the inactive A/B slot, health checks decide whether the new slot stays or the node rolls back.

Two trust chains meet here. The boot chain (signed bootloader, signed kernel, TPM measurement) guarantees the node runs only what I signed. The update chain (offline signing, per-device policy, A/B slots with automatic fallback) guarantees I can change what it runs without ever being able to brick it. A node that misses a campaign because its link is down simply picks the bundle up when the link returns, the same DDIL store-and-forward pattern as my field nodes.

Measured Boot vs Secure Boot

Steps 01 through 04 implement two different ideas that are easy to conflate. Secure boot is a gate: the bootloader checks signatures and refuses to execute anything unsigned, so bad code never runs. Measured boot is a record: each boot stage hashes the next and extends the hash into a TPM Platform Configuration Register (PCR), so at the end the TPM holds a fingerprint of exactly what booted, and the disk keys are sealed to that fingerprint. Tamper with the image and the PCRs change; the TPM then refuses to unseal, and the attacker gets a brick, not the data.

Test Status

A quick visual of where the six tests stand, drawn straight from the targets table below. This is a status snapshot, not results: "passing" means the mechanism fires correctly on the bench rig (T1, T2), not that the run counts are statistically meaningful. Full numbers land only when the campaign, rollback, cold-matrix, and chaos batteries have accumulated enough runs.

Wiring & Connections

The hardware side of this build is deliberately boring, most of the security lives in software. What is physically connected, and why:

How I Built It

# device key lives on the offline machine only
openssl genrsa 2048 > device-private.pem
openssl rsa -in device-private.pem -pubout > device-public.pem

# boot.sig = raw binary RSA-2048 signature over sha256(boot.img), not hex
openssl dgst -sign device-private.pem -keyform PEM -sha256 \
  -out boot.sig work/boot.img

# EEPROM config enforcing signed boot
grep SIGNED_BOOT boot.conf
#   SIGNED_BOOT=1

# flash EEPROM and program the pubkey hash into OTP - ONE WAY, no undo
cd usbboot && sudo ./rpiboot -d secure-boot-msd

# /boot/firmware/config.txt
dtparam=spi=on
dtoverlay=tpm-slb9670

# after reboot, the kernel should bind the driver
dmesg | grep -i tpm
#   tpm_tis_spi spi0.0: 2.0 TPM (device-id 0x1B, rev-id 22)
ls -l /dev/tpm0 /dev/tpmrm0
sudo tpm2_getcap properties-fixed | head -5

# PCRs after a clean boot
sudo tpm2_pcrread sha256:0,1,2,3,7

# the event log: which component produced which measurement
sudo tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements | less

# baseline for comparison after updates
sudo tpm2_pcrread sha256:0,1,2,3,7 > /etc/fleet/pcr-baseline.txt

sudo cryptsetup luksFormat --type luks2 /dev/mmcblk0p3
sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+7 /dev/mmcblk0p3

# /etc/crypttab
#   rootfs  /dev/mmcblk0p3  none  tpm2-device=auto,discard

# confirm the TPM2 token is enrolled
sudo cryptsetup luksDump /dev/mmcblk0p3 | grep -A3 tpm2

# generate the key ON the card (OpenPGP application, ed25519)
gpg --edit-card
#   gpg/card> admin
#   gpg/card> generate      # key type: ed25519

# sign an update manifest; the PIN pads the card, the card signs
gpg --detach-sign --armor --local-user FLEET-ROOT manifest.json
gpg --verify manifest.json.asc manifest.json

# offline build machine
syft dir:./rootfs -o spdx-json=update-1.4.0.spdx.json
cosign sign-blob --key "pkcs11:object=FLEET-ROOT" \
  --bundle update-1.4.0.cosign.bundle update-1.4.0.raucb

# on the node: verify BEFORE install; && is the gate
cosign verify-blob --key /etc/fleet/fleet-root.pub \
  --bundle update-1.4.0.cosign.bundle update-1.4.0.raucb \
  && rauc install update-1.4.0.raucb

[system]
compatible=tyfsadik-fleet-node
bootloader=custom

[keyring]
path=/etc/rauc/fleet-root-ca.cert.pem

[slot.rootfs.0]
device=/dev/mmcblk0p2
type=ext4
bootname=A

[slot.rootfs.1]
device=/dev/mmcblk0p3
type=ext4
bootname=B

# offline machine: build the signed bundle
rauc bundle --cert=fleet-rel.cert.pem \
  --key="pkcs11:object=FLEET-ROOT;type=private" \
  build/ update-1.4.0.raucb

#!/bin/sh
# runs once on the first boot of a new slot
systemctl is-active --quiet rauc fleet-heartbeat || exit 1
tpm2_pcrread sha256:7 -Q || exit 1
ping -c1 -W2 fleet.tyfsadik.lan >/dev/null || exit 1
rauc status mark-good

# fleet-health.service: Type=oneshot, After=network-online.target,
# ExecStart=/usr/local/sbin/fleet-health-check, WantedBy=multi-user.target

# campaign.yaml on the fleet server
campaign: rover-fleet-2026-09
image: update-1.4.0.raucb
targets: [rover-01, rover-02]
policy:
  rollout: serial        # one node at a time
  require_healthy: true  # wait for mark-good before the next node
  resume: true           # resume partial downloads

# node-side enrolment (my small script over the fleet CA)
sudo fleet-enrol --server https://fleet.tyfsadik.lan \
  --tpm --out /etc/fleet/device.json

# node-side fetcher: resume, retry, then check against the signed digest
curl -C - --retry 20 --retry-all-errors \
  -o update-1.4.0.raucb \
  https://fleet.tyfsadik.lan/bundles/update-1.4.0.raucb
echo "$DIGEST  update-1.4.0.raucb" | sha256sum -c -

# rehearse DDIL conditions on the bench
sudo tc qdisc add dev eth0 root netem loss 15% delay 400ms
sudo tc qdisc del dev eth0 root   # back to normal

while true; do
  T=$(cat /sys/bus/w1/devices/28-*/w1_slave | awk -F= '/t=/{print $2/1000}')
  echo "$(date -Is) temp=${T}C slot=$(rauc status --output-format=shell \
    | grep ^RAUC_SYSTEM_BOOTED)" >> /var/log/cold-update.log
  sleep 30
done

# corrupt 1 MiB in the middle of a bundle mid-transfer
dd if=/dev/urandom of=update-1.4.0.raucb bs=1M seek=40 count=1 conv=notrunc

# power-cut mid-flash at a random point in the write
sleep $((RANDOM % 90))
curl -X POST "http://plug.local/relay/0?turn=off"

# after power returns: must show a GOOD, bootable slot
rauc status --output-format=shell | grep -E "BOOTED|GOOD"

Testing & Metrics

The test protocol treats trust properties as measurable, pass/fail things. Each run below has a fixed setup, procedure, and metric; results are logged per run so the eventual dataset is counts, not vibes.

T1, Boot-chain rejection

• Setup: OTP-programmed node, serial console attached to catch early-boot output.
• Procedure: attempt to boot (a) an unsigned image, (b) an image signed with the wrong key, (c) a valid image with one byte flipped.
• Metric: boots accepted. Target: 0 of 3, every variant must be refused before the kernel starts.

T2, TPM seal integrity

• Setup: node with LUKS rootfs, key sealed to PCRs 0+7, recovery passphrase in a separate keyslot.
• Procedure: boot a deliberately modified initramfs; attempt to unlock the disk. Repeat with the unmodified image as control.
• Metric: unseal blocked for the tampered image, unseal succeeds for the control, recovery passphrase works. Target: 3 of 3.

T3, Update campaign reliability

• Setup: two enrolled nodes, fleet server running a serial-rollout campaign.
• Procedure: run repeated full campaigns (enrol check, transfer, verify, install, trial boot, health gate, mark-good) at room temperature.
• Metric: per-stage success counts across campaigns. Target: 100% mark-good on every node, serial ordering respected every time.

T4, Forced rollback

• Setup: a deliberately crippled image (network service masked so the health gate can never pass).
• Procedure: push it as a campaign; observe trial boot, health-gate failure, boot-count expiry, fallback.
• Metric: campaigns ending on the last-known-good slot with no manual intervention. Target: 100%.

T5, Cold matrix

• Setup: node plus DS18B20 probe in the chest freezer, logging temperature and slot state together.
• Procedure: run T3 and T4 at room temperature, at freezer temperature (~-20 °C), and during warm-up.
• Metric: success vs temperature, plus any cold-only errors (slow flash, marginal reads). Target: identical outcomes across the matrix; deviations get root-caused, not averaged away.

T6, Fault injection (chaos)

• Setup: smart plug on the node's power, corruption script for mid-transfer damage, invalid-signature bundles.
• Procedure: inject each fault repeatedly, warm and cold: truncated bundles, flipped payload bytes, power cuts at random offsets into the flash write.
• Metric: returns to a signed, working state with zero manual recovery. Target: 100% across every fault type and temperature.

Targets and current status

Full numbers (campaign counts, rollback rates, cold-vs-warm comparison) will be published once the matrix has accumulated enough runs to mean something. Nothing above is a final result yet; "passing" means the mechanism fires correctly on the bench, not that the run count is statistically meaningful.

Troubleshooting & Lessons

• TPM never enumerated. The module sat silently on the header with no /dev/tpm0. The scope showed no clock on SCLK, SPI itself was off, and my first overlay name didn't match the chip. Fix: dtparam=spi=on plus the correct dtoverlay=tpm-slb9670 for the SLB9670-class board, confirmed with dmesg | grep -i tpm. Lesson: debug the bus before blaming the chip.
• I locked my own node out of its disk. A legitimate kernel update changed the PCR measurements, the TPM refused to unseal, and the node couldn't decrypt its own rootfs after a perfectly valid update. Fix: the offline recovery passphrase (separate LUKS keyslot, stored offline), and re-sealing the TPM policy is now an explicit step in the update pipeline, run with every build.
• OTP is one-way, and I knew it could still surprise me. Programming the secure-boot key hash into OTP cannot be undone, a mistake there turns a Pi into a paperweight. Fix: rehearsed the entire sign-flash-boot cycle three times on a second, un-programmed board first, and only burned OTP after three clean reject-and-accept cycles.
• A cheap SD card died during power-cut rehearsals. Random mid-flash power cuts are exactly what consumer cards hate; the card, not the update mechanism, was the weak point. Fix: high-endurance cards from the Phase 0 kit, and the A/B layout means even a failed slot write never touches the running system.
• Condensation in the freezer. The first cold run ended with a damp node that I was not about to power up. Fix: the node now goes into and out of the freezer in a sealed bag and warms to room temperature before power-up, the same discipline as never charging a cold battery. Conformal coating is on the list for nodes that will live outdoors.

Status & Roadmap

• Done: hardware on the bench (TPM board, both YubiKeys, fleet nodes); secure-boot chain with signed bootloader and kernel; TPM enumerated and measured boot logging; LUKS keys sealed to PCRs (with an offline recovery key after the lockout lesson); basic A/B update with a health-gate script; first cold rollback runs in the freezer; mid-transfer corruption verified to reject.
• In progress: automated mark-good flow end to end; per-device enrolment and serial-rollout campaign policies on the homelab fleet server; store-and-forward delivery under netem degraded-link rehearsal; accumulating the cold-temperature test matrix (T5).
• Next: full power-cut chaos battery (T6) warm and cold; publish the test dataset (campaign counts, rollback rate vs temperature); conformal coating for outdoor nodes; extend enrolment to every rover and sensor node in the program.

Related Builds

• Canadian Defence Hardware Program, the hub: every build, the full phased BOM, and the Canadian requirements each one answers.
• Zero-Trust Air-Gapped Supply Chain, the signing, SBOM, and provenance discipline this fleet inherits, proven on containers first.
• DDIL-Proof DevOps, the store-and-forward delivery patterns the fleet server reuses for nodes on degraded links.
• Cold-Soak Characterization Rig, the shared chest freezer and instrumentation behind this build's cold-chamber update tests.

Build Narrative: Phase by Phase

Phase 1: TPM bringup. An SLB9670-class TPM 2.0 add-on board on the Pi's SPI header. Bringup fought the bus, not the chip: dtparam=spi=on plus the correct dtoverlay=tpm-slb9670, confirmed with dmesg | grep -i tpm and /dev/tpm0 appearing. The scope showed no clock on SCLK until the overlay matched.

Phase 2: secure boot chain. A signed bootloader and kernel chain from the bootloader up, with the TPM 2.0 measuring each stage. The OTP key-hash programming was rehearsed three times on a second, un-programmed board, with full reject-and-accept cycles, before being burned on the real node. OTP is one-way: a mistake there is a paperweight.

Phase 3: sealed disk keys. LUKS disk keys sealed to the TPM's PCR measurements so the keys are held hostage to the measured boot state, plus a separate offline recovery passphrase in its own LUKS keyslot. That recovery key exists because a legitimate kernel update once changed the PCRs and locked the node out of its own disk; re-sealing the TPM policy is now an explicit step in the update pipeline.

Phase 4: signing ceremony. Update images are signed with Ed25519 keys that live on two YubiKeys (one is the backup), offline, never on a networked machine. The fleet nodes verify every image against the pinned key before they will install it.

Phase 5: A/B updates with rollback. A/B partitions with a health-gate script and a mark-good flow: a bad image rolls itself back to the known-good slot. Mid-transfer corruption was verified to reject, and the first cold rollback runs happened in the freezer. Power-cut rehearsals exposed a weak point that was not the mechanism at all: a cheap SD card dying mid-flash, replaced with high-endurance cards from the Phase 0 kit.

What You Need

The fleet nodes are shared with the rover and sensor builds, so the incremental cost of trust is essentially the TPM and the keys. The scarcest resource is the discipline to rehearse destructive operations on a spare board.

Challenges and Lessons Learned

• The TPM never enumerated. The module sat silently on the header with no /dev/tpm0. Lesson: debug the bus before blaming the chip, the overlay name and SPI enable were the whole problem.
• I locked my own node out of its disk. A legitimate kernel update changed the PCR measurements and the TPM refused to unseal. Lesson: keep an offline recovery passphrase in a separate LUKS keyslot, and make re-sealing the TPM policy an explicit step in the update pipeline.
• OTP is one-way, and it can still surprise you. Burning the secure-boot key hash into OTP cannot be undone. Lesson: rehearse the entire sign-flash-boot cycle three times on a spare board, and only burn OTP after clean reject-and-accept cycles.
• A cheap SD card died during power-cut rehearsals. Random mid-flash power cuts are exactly what consumer cards hate. Lesson: high-endurance cards, and the A/B layout means a failed slot write never touches the running system.
• Condensation in the freezer. The first cold run ended with a damp node I would not power up. Lesson: sealed-bag discipline in and out of the cold, warm to room temperature before power-up; conformal coating is on the list for outdoor nodes.

What Is Next

• Automated mark-good flow end to end, removing the manual step from the update pipeline.
• Per-device enrolment and serial-rollout campaign policies on the homelab fleet server.
• Store-and-forward update delivery under netem degraded-link rehearsal, because field nodes update over bad links.
• Complete the cold-temperature test matrix (T5) and run the full power-cut chaos battery warm and cold (T6).
• Publish the test dataset: campaign counts and rollback rate versus temperature.
• Conformal coating for outdoor nodes, then extend enrolment to every rover and sensor node in the program.