Multi-Domain Common Operating Picture

Before a node is allowed to say anything, it gets an identity. Each node has its own Ed25519 keypair, generated once at provisioning time on the bench laptop, never on the node, so the entropy source is a real machine. The secret key is flashed into the node's firmware image or dropped into its config partition; the public half goes into a registry file that the tracker, the audit log, and any future consumer load at startup. The payoff is that any detection can be traced to the exact node that signed it, and a cloned or spoofed node fails verification instead of poisoning the picture. The known weakness: a physically captured node yields its key, so the hardened path is a secure element (ATECC608 class) or TPM-backed storage where the key signs without the secret ever leaving hardware, bench target, not yet built.

BASH, provisioning a new node keypair (run once per node)

python3 - <<'EOF'
from nacl.signing import SigningKey
import pathlib
node = "seismic-node-03"
sk = SigningKey.generate()
d = pathlib.Path(f"keys/{node}")
d.mkdir(parents=True, exist_ok=True)
(d / "secret.key").write_bytes(bytes(sk))
(d / "public.hex").write_text(sk.verify_key.encode().hex())
print(node, sk.verify_key.encode().hex())
EOF
chmod 600 keys/seismic-node-03/secret.key

How I verified: I signed a known payload with the secret key and confirmed nacl.signing.VerifyKey accepts it, then flipped one byte of the public key and confirmed verification now fails. What can go wrong here is boring and fatal: shipping the same key to two nodes, which makes attribution meaningless, the registry file is checked into Git so duplicates show up in review.

The bus is a single Mosquitto broker on the Pi 5, with its persistence directory on the 256 GB NVMe so retained messages and queued QoS 1 traffic survive reboots without eating an SD card. Every gateway authenticates, and an ACL pins each one to its own topic subtree, the seismic gateway physically cannot publish as a hydrophone node. I evaluated Zenoh as the alternative transport: its built-in store-and-forward and peer-to-peer mode are genuinely attractive for DDIL links, but MQTT's tooling, broker maturity, and the fact my existing services already speak it won the first iteration. Zenoh stays on the bench as the candidate for a brokerless version.

MOSQUITTO, /etc/mosquitto/conf.d/cop.conf

listener 1883 0.0.0.0
protocol mqtt
allow_anonymous false
password_file /etc/mosquitto/passwd
acl_file /etc/mosquitto/cop.acl
persistence true
persistence_location /var/lib/mosquitto/   # mounted on the NVMe
max_queued_messages 10000
log_type error
log_type warning

ACL, /etc/mosquitto/cop.acl (one gateway, one subtree)

user gw-seismic
topic write cop/ground/seismic/#
topic read cop/sys/time

user gw-hydro
topic write cop/water/hydrophone/#
topic read cop/sys/time

How I verified: mosquitto_pub as gw-seismic into cop/water/... is rejected, and mosquitto_sub -t 'cop/#' -v from the tracker account sees every domain. The classic failure is forgetting persistence_location points at the SD card; I confirmed the mount with df -h /var/lib/mosquitto before load-testing.

Every detection, from every domain, is the same small JSON object: node_id, a per-node monotonic seq counter (the outbox row id on Pi-class nodes), ISO-8601 timestamp, detection class, a bearing or position estimate with its covariance, a confidence value, and the Ed25519 signature over the canonicalized bytes. Canonicalization matters: keys are sorted and whitespace stripped before signing, because a re-serialized payload must verify byte-for-byte, and the signature is computed over the object without the sig field, which is attached afterwards. Dedup keys on the (node_id, seq) pair, not (node_id, ts), because two detections from one node can share a timestamp tick. The signature is produced on the node before the payload touches the radio, so the signature covers the original timestamp, which is what lets the tracker trust late, replayed data after an outage.

JSON, one detection on the wire

{"bearing_deg":214,"class":"vehicle","confidence":0.82,
 "cov":[3.1],"node_id":"seismic-node-03","seq":4821,
 "ts":"2026-09-19T14:03:22.184Z",
 "sig":"Zk9...base64..."}

BASH, signing and publishing one detection (QoS 1)

payload=$(python3 sign_detection.py \
  --node seismic-node-03 --class vehicle --bearing 214 --conf 0.82)
mosquitto_pub -h broker.lan -u gw-seismic -P "$GW_PASS" \
  -t cop/ground/seismic/seismic-node-03 -q 1 -m "$payload"

How I verified: the tracker verifies every message with PyNaCl on arrival; I hand-edited one byte of a logged payload and confirmed it is rejected and counted in a signature_failures_total metric. What goes wrong: float re-serialization differences between MicroPython and CPython breaking canonical form, solved by fixing the field order and precision in the schema, not in each signer.

Multi-sensor fusion is meaningless without aligned timestamps, a 200 ms skew between a seismic node and an acoustic node can flip which detection the tracker thinks came first, and bearing-only fusion is even more sensitive. Each aggregator node carries a u-blox M9N module; its pulse-per-second output lands on a Pi GPIO, and gpsd hands both the NMEA sentences and the PPS edge to chronyd, which disciplines the system clock. NMEA alone has hundreds of milliseconds of jitter, so NMEA names the second and PPS marks its start.

CONFIG, /boot/firmware/config.txt and gpsd unit on the aggregator Pi

# /boot/firmware/config.txt
enable_uart=1
dtoverlay=pps-gpio,gpiopin=18

# /etc/default/gpsd
DEVICES="/dev/ttyAMA0 /dev/pps0"
GPSD_OPTIONS="-n"

CONFIG, /etc/chrony/chrony.conf (NMEA names the second, PPS marks it)

refclock SHM 0 refid NMEA precision 1e-1 offset 0.0 delay 0.2 noselect
refclock PPS /dev/pps0 refid PPS lock NMEA prefer
makestep 1.0 3

How I verified: ppstest /dev/pps0 shows a clean once-per-second assert, and chronyc sources -v shows the PPS refclock selected (*) with NMEA locked but not selected. ESP32 mesh nodes without their own GNSS inherit time from their gateway over LoRa and stamp detections with both their local tick and the gateway-corrected estimate, so the tracker can weight them accordingly. What goes wrong: chrony locking onto NMEA instead of PPS, see the troubleshooting section, because it happened to me.

This is the DDIL pattern from my DDIL-Proof DevOps work applied to physical nodes. Every node keeps a local outbound queue: a SQLite database in WAL mode on the Pi-class nodes, a ring buffer in flash on the ESP32s. A detection is written to the outbox first, then published with MQTT QoS 1, and only marked acknowledged when the PUBACK arrives. When the link drops, rows pile up; when it returns, the node replays the backlog in ID order, and the broker sees late data clearly marked by its original signed timestamps.

SQL, the outbox schema on every Pi-class node

PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
CREATE TABLE IF NOT EXISTS outbox (
  id         INTEGER PRIMARY KEY AUTOINCREMENT,
  topic      TEXT    NOT NULL,
  payload    BLOB    NOT NULL,
  created_ms INTEGER NOT NULL,
  acked      INTEGER NOT NULL DEFAULT 0
);

PYTHON, the replay loop (runs whenever the broker is reachable)

rows = db.execute(
    "SELECT id, topic, payload FROM outbox "
    "WHERE acked = 0 ORDER BY id LIMIT 100").fetchall()
for row_id, topic, payload in rows:
    info = client.publish(topic, payload, qos=1)
    info.wait_for_publish()          # returns after PUBACK
    db.execute("UPDATE outbox SET acked = 1 WHERE id = ?", (row_id,))
db.commit()

How I verified: I pulled the Ethernet cable for two minutes while a node fired test detections, reconnected, and confirmed via mosquitto_sub that every message arrived, in ID order within each replay batch, with original timestamps. QoS 1 is at-least-once, not exactly-once, so a few duplicates did arrive and were dropped by the tracker's (node_id, seq) unique index: the no-loss guarantee comes from replay, the no-duplicate guarantee comes from the consumer's idempotency, and both halves need to be in place. The failure mode to design for is a full SD card mid-outage, which is why the queues live on NVMe/high-endurance media and the schema caps retained rows.

The seismic mesh nodes are ESP32s on 915 MHz LoRa, they cannot reach the broker directly, so a gateway Pi terminates the LoRa link and re-publishes their frames to MQTT. The ESP32 signs the detection with its own Ed25519 key using the Arduino Crypto library, then ships one compact frame: payload, signature, and a CRC. The gateway checks the CRC, stamps the gateway-corrected time next to the node's own estimate, and publishes to the node's topic under the gateway's ACL. The signature still belongs to the node, the gateway is a courier, not an author, and a compromised gateway can drop or delay messages but cannot forge them.

ARDUINO, signing on the ESP32 before the LoRa transmit

#include <Crypto.h>
#include <Ed25519.h>

uint8_t sig[64];
Ed25519::sign(sig, secretKey, publicKey,
              payload, payloadLen);
// frame = [payloadLen | payload | sig | crc16] -> LoRa transmit

PYTHON, the gateway bridge, receive side

frame = lora.recv(timeout=5)
if frame and crc16(frame[:-2]) == struct.unpack("<H", frame[-2:])[0]:
    det = parse_frame(frame)
    det["gw_ts"] = iso_now()          # PPS-disciplined gateway clock
    client.publish(f"cop/ground/seismic/{det['node_id']}",
                   json.dumps(det), qos=1)

How I verified: I corrupted one byte of a frame in firmware and confirmed the gateway drops it (CRC) and logs the event rather than bridging garbage. What goes wrong: LoRa's low duty cycle means a backlog replay can starve new detections, so the gateway round-robins between old and fresh frames instead of draining oldest-first blindly.

The tracker is a Python service on the broker Pi that subscribes to cop/+/+/+, verifies signatures, and maintains a track table. Each track is a constant-velocity Kalman filter over position and velocity (built on FilterPy). Air detections come in from the passive radar node and the acoustic drone detector, water contacts from the hydrophone monitor, and ground activity from the seismic mesh; every detection arrives with its own covariance, so the filter can weigh a sharp bearing cut against a noisy position fix honestly. Association is gated by Mahalanobis distance: a detection attaches to the nearest track inside the gate, or starts a tentative new track that must be confirmed by a second hit before it appears on the map.

PYTHON, the per-track filter (FilterPy)

from filterpy.kalman import KalmanFilter
from filterpy.common import Q_discrete_white_noise
import numpy as np

kf = KalmanFilter(dim_x=4, dim_z=2)      # state: x, y, vx, vy
kf.F = np.array([[1,0,1,0],
                 [0,1,0,1],
                 [0,0,1,0],
                 [0,0,0,1]])             # constant velocity, dt=1s
kf.H = np.array([[1,0,0,0],
                 [0,1,0,0]])             # we observe position only
kf.P *= 100.0                            # big initial uncertainty
kf.R = det_cov                           # from the detection itself
kf.Q = Q_discrete_white_noise(dim=2, dt=1.0, var=0.5, block_size=2,
                               order_by_dim=False)   # match the [x, y, vx, vy] state order;
                                                     # the default True gives [x, vx, y, vy]

PYTHON, Mahalanobis gating before association

S = kf.P[:2,:2] + kf.R
# scipy's mahalanobis needs 1-D vectors, takes the INVERSE covariance, and returns d, not d-squared
d = mahalanobis(np.asarray(det_xy).ravel(), kf.x[:2].ravel(), np.linalg.inv(S))
if d**2 < 9.21:       # chi2 gate, 2 dof, 99%
    kf.update(det_xy)  # attach to this track
else:
    spawn_tentative(det)  # needs a second hit to go firm

The output is a single track table, track ID, state estimate, uncertainty ellipse, contributing sensors, and a confidence score, published back onto the bus at cop/fused/tracks so the dashboard and the audit log both consume the same truth. How I verified: I replayed logged detections from two bench nodes watching the same walked perimeter path and confirmed they associate into one track instead of two; what goes wrong is over-tight gating splitting one target into many tentative tracks, which is why association tuning is still in progress.

The COP itself is a Leaflet map served from the homelab Kubernetes cluster, showing air, water, and ground tracks with confidence scores and uncertainty ellipses, next to a live alert queue and per-node signal panels. A small FastAPI backend subscribes to cop/fused/tracks and the domain topics and pushes updates to the browser over a websocket, so the map updates without polling. Grafana covers the plumbing health behind it: broker queue depth, outbox backlog per node, chrony offsets, and node last-seen times. The design constraint is deliberate: the system cues, it never acts, there is no automated response path, and every alert ends at a person.

YAML, the dashboard deployment on the homelab cluster

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cop-dashboard
spec:
  replicas: 1
  selector:
    matchLabels: { app: cop-dashboard }
  template:
    metadata:
      labels: { app: cop-dashboard }
    spec:
      containers:
        - name: web
          image: registry.lan/cop-dashboard:0.4.2
          ports:
            - containerPort: 8080
          env:
            - name: MQTT_HOST
              value: broker.lan

How I verified: with a node firing test detections, I watch the track appear on the map within a couple of seconds and the Grafana last-seen panel tick with it; kubectl rollout status deploy/cop-dashboard after each image bump keeps the deploy honest. What goes wrong: a reconnecting node replaying a two-minute backlog can flood the websocket and freeze the map, fixed by server-side batching of replay bursts, the hard way.

Every alert carries its full evidence chain: the raw detection payloads it was fused from, the node signatures that authenticate them, the tracker version and parameters that produced it, and the operator's disposition. All of it is appended to the tamper-evident log from my Decision Black Box build, a hash-chained, append-only file where each line embeds the SHA-256 of the previous line, so any silent edit, deletion, or reorder breaks the chain and is detectable on audit.

PYTHON, hash-chained append (the whole trick is prev_hash)

import hashlib, json, os

def append_alert(log_path, record):
    prev = b""
    if os.path.exists(log_path) and os.path.getsize(log_path):
        prev = open(log_path, "rb").read().strip().split(b"\n")[-1]
    record["prev_hash"] = hashlib.sha256(prev).hexdigest() if prev else "0" * 64
    with open(log_path, "ab") as f:
        f.write(json.dumps(record, sort_keys=True).encode() + b"\n")

How I verified: a verifier script re-walks the chain and recomputes every prev_hash; I edited one historical line with sed and the verifier flags exactly that line and everything after it. One honest limit: a hash chain catches edits to sealed history but cannot catch the newest tail being deleted, so the latest line hash is published to the broker on a schedule and the verifier checks the tail against that anchor. The one that bites in practice is log rotation breaking the chain across files, so rotation renames only completed, sealed files and the new file's first prev_hash points at the last line of the sealed one.

The acceptance test for this system is failure. The chaos harness is a set of scripts that cut links with tc netem, restart the Mosquitto container and kill the tracker pod to watch the system recover, pull power on individual nodes, and corrupt one node's clock to watch the tracker down-weight its skewed detections instead of poisoning the picture. The two metrics that matter are time-to-recover (how long until a domain is back on the map after its link or node dies) and track continuity (whether the same physical target keeps the same track ID across the outage).

BASH, the chaos toolkit

# degrade the link to the broker: 40% loss, 800ms +/- 200ms delay
sudo tc qdisc add dev eth0 root netem loss 40% delay 800ms 200ms
sleep 120
sudo tc qdisc del dev eth0 root

# kill the tracker; kubernetes reschedules it
kubectl delete pod -l app=cop-tracker

# corrupt one node's clock by +47 seconds
ssh node3 'sudo date -s "+47 seconds"'

How I verified: the harness records broker timestamps, outbox depths, and track IDs before/during/after each scenario into a run directory, so every change to the system can be regression-tested against the same failure matrix. What can go wrong: forgetting to delete the tc qdisc after a run and then "discovering" a phantom performance bug, the harness now asserts a clean qdisc on exit.