Decision Black Box
A tamper-evident audit layer for human-in-the-loop decisions in a simulated airspace-deconfliction scenario: a hash-chained, signed, append-only log in the style of certificate transparency, a replay tool that reconstructs any decision state, and OPA policy-as-code checks on every entry.
Overview
When a decision support system recommends something and a human approves it, someone has to be able to answer afterwards: what did the operator see, what did the system recommend, which policy version was in force, and who signed off? This project is a flight recorder for decisions, built around a fictional airspace-deconfliction simulator. Every recommendation, override, and approval is written to an append-only log that cannot be edited or quietly reordered after the fact.
The core is a hash-chained, signed log modelled on certificate transparency (RFC 6962): each entry includes the hash of the previous one, the log operator signs Merkle tree heads, and auditors can request inclusion and consistency proofs. On top of that sit two tools: a replay engine that rebuilds the exact state of the simulated airspace picture at any entry in the log, and OPA policy-as-code checks that evaluate every decision against the rules of engagement simulated for the scenario (for example: no deconfliction approval without two-operator confirmation).
Why bother: accountability and responsible AI are treated seriously in Canadian defence circles, and DND/CAF has published AI strategy commitments around traceability and human oversight. I wanted to practice the engineering behind those commitments. Everything here runs on simulated data I generate myself. There is no weapons logic, no real targeting information, and nothing classified anywhere in this project.
Architecture
(fictional tracks, Python)"] REC["Decision support stub
(rule-based recommender)"] end subgraph Audit["Audit Layer"] API["Ingestion API
(Go, HTTP + gRPC)"] LOG[("Append-only log
hash chain + Merkle tree")] SIGN["Signer
(Ed25519, key in KMS-style seal)"] OPA["OPA sidecar
(Rego policies)"] end DB[("Event store
(PostgreSQL, WORM semantics)")] STH[("Signed tree heads
gossip endpoint")] RPL["Replay tool
rebuild state at entry N"] DASH["Audit dashboard
(proofs, policy verdicts)"] SIM --> REC REC --> API SIM --> API API --> OPA OPA --> API API --> LOG LOG --> SIGN SIGN --> STH LOG --> DB DB --> RPL STH --> DASH DB --> DASH
Figure 1: simulated sources feed the ingestion API; every event is policy-checked, chained, signed, and stored before it can be replayed or audited.
The ingestion API is the only writer. An event (sensor observation, recommendation, operator decision) is validated, sent to the OPA sidecar for a policy verdict, then appended to the log. The append is atomic with the hash-chain link: the new entry commits the SHA-256 of its canonical JSON encoding plus the previous entry's hash, so any edit to history invalidates every subsequent link. Periodically the signer folds the log into a Merkle tree, signs the tree head with an Ed25519 key (Ed25519 public keys are 32 bytes and signatures are 64 bytes, per RFC 8032), and publishes it, which is what lets an outside auditor verify the log without trusting me.
Hash-Chain Verification and Replay
for entry N"] --> B["Fetch Merkle inclusion proof
entry N vs signed tree head"] B --> C{"Hash path recomputes
to signed root?"} C -->|no| BAD["TAMPER: entry or tree
does not match signed head"] C -->|yes| D["Fetch signed tree heads
T_old and T_new"] D --> E{"Consistency proof valid?
(T_old is a prefix of T_new)"} E -->|no| BAD2["TAMPER: history was rewritten
or forked"] E -->|yes| OK["Log verified:
append-only, untampered"] OK --> R["Replay: fold events 1..N
through the state machine"] R --> S["Reconstructed decision state:
tracks, recommendations, approvals,
policy verdicts at that moment"]
Figure 2: verification proves the log is append-only before replay reconstructs the state at any entry.
- Inclusion proof: an auditor gets the Merkle path for a single entry and checks it against a signed tree head, the same proof a certificate transparency monitor runs.
- Consistency proof: two signed tree heads are compared to prove the log only ever grew; this is the property that catches silent history rewrites.
- Replay: events are folded in order through a deterministic state machine, so state at entry N is always reproducible. The same code path backs the audit dashboard and any after-action review.
How the tamper-evidence chain works (and its limits)
Every entry carries the hash of the previous entry, so altering any record breaks every link after it, and the Merkle tree over the log lets an auditor verify a single entry against a signed tree head with a short inclusion proof. Consistency proofs show that a newer tree head is a strict extension of an older one, which is what catches a silent history rewrite or a fork. This is the Certificate Transparency pattern: append-only Merkle logs with signed tree heads make tampering detectable, not impossible. A verifier that nobody runs is decoration, which is why the rescan verifier exists and why the gossip endpoint is on the roadmap: the protection comes from independent parties comparing tree heads, not from the hash function alone.
How OPA gates an action
When a write path needs a policy decision, the service hands OPA a JSON input describing the proposed action: who requested it, what resource it touches, and the current context. OPA evaluates the Rego policy against that input and returns a JSON verdict, allow or deny, with the reasons. The service enforces the verdict and writes both the request and the verdict into the log, so the policy decision itself becomes auditable evidence. The first policy on the roadmap is two-person approval: an action above a threshold needs two distinct authorised identities, and OPA is what checks that before anything is written.
Stack
| Layer | Technology | Why |
|---|---|---|
| Audit log core | Go (stdlib crypto, x/crypto Ed25519) | Static binaries, strong typing for a component that must not be ambiguous |
| Transparency design | RFC 6962-style Merkle tree (Trillian-inspired) | Battle-tested inclusion and consistency proofs |
| Policy as code | Open Policy Agent (Rego), sidecar over HTTP | Policies versioned in Git, evaluated per decision, independently auditable |
| Event store | PostgreSQL (append-only table, no UPDATE/DELETE grants) | WORM semantics with boring, well-understood tooling |
| Simulation | Python scenario generator (fictional tracks) | Deterministic seeded runs so reviews are reproducible |
| Deployment | Docker Compose on my homelab, Terraform later | Whole stack reproducible from Git |
How I'm Building It
A versioned JSON schema for the three event types (observation, recommendation, decision), stored in PostgreSQL with a role that physically cannot run UPDATE or DELETE. Inserts compute the chain hash in a trigger-checked transaction.
A Go service that folds pending entries into a Merkle tree every N seconds, signs the tree head with an Ed25519 key held behind a software seal, and publishes signed heads to a gossip-style endpoint so forks are detectable.
Rego policies evaluate every decision event before it is committed. Current example: a simulated deconfliction approval requires two distinct operator identities within a time window. The verdict (allow/deny plus policy version) is logged alongside the event, so policy changes are visible in history.
A seeded Python generator produces fictional aircraft tracks, the rule-based stub issues recommendations (re-route, hold, handoff), and scripted operators approve or override. Deterministic seeds mean any reviewer can reproduce an incident exactly.
A CLI (replay --entry N) folds events through the state machine and renders the picture at that moment; a small web UI shows the log, proof status, and policy verdicts per decision.
How Defence Could Implement This
The accountability pattern here is deliberately generic, which is what makes it transferable to real CAF and NORAD contexts:
- Command and control audit: any C2 system that surfaces machine recommendations (air tasking, track classification aids, logistics triage) could write the same three event types to a tamper-evident log, giving boards of inquiry and after-action reviews a trustworthy record of who saw what and decided what.
- Responsible AI assurance: DND/CAF AI strategy commitments call for traceable, human-accountable AI. OPA-style policy gates are a concrete mechanism: rules like "human confirmation required" stop being documentation and become enforced, logged code.
- Cross-network integrity: signed tree heads are small and one-way friendly. A consistency proof can cross a classification boundary where the full log cannot, so a higher-side auditor can verify a lower-side log was not rewritten without the data itself moving.
- Multi-party verification: in NORAD or coalition settings, each party can hold signed tree heads from the others, so no single operator can quietly edit a shared decision record.
- Domestic operations: the same pattern applies outside combat contexts, for example Operation LENTUS tasking, where provincial and military decisions interleave and a clean audit trail matters for public accountability.
None of this requires new sensors or new platforms. It is an integrity layer over systems that already exist, and it can be prototyped at homelab scale, which is exactly what I am doing with simulated data.
How to Build This Yourself (step-by-step guide)
- Define your events first. Pick three event types for your scenario (I use observation, recommendation, decision) and write a versioned JSON schema before any storage code.
- Make the store append-only for real. In PostgreSQL, create a writer role with INSERT and SELECT only, no UPDATE, DELETE, or TRUNCATE. Test that deletion fails; do not assume it.
- Add the hash chain on insert. Each row stores the SHA-256 of its canonical encoding plus the previous row's hash. Verify with a full-table rescan that recomputes every link.
- Read RFC 6962, then build the Merkle tree. Implement leaf hashing with domain separation prefixes, fold entries into a tree, sign the tree head with Ed25519, and expose inclusion and consistency proof endpoints.
- Write one OPA policy early. A single Rego rule (for example, two-person approval) wired into the write path teaches you the whole pattern. Log the verdict and policy version with every event.
- Build replay as a fold, not a query. State at entry N is a pure function of events 1 through N. Deterministic seeds in your simulator make replay output reproducible for reviewers.
- Attack your own log. Write a test that edits a middle row, one that deletes rows, and one that forks the tree. All three must be caught by verification, or the design is decoration.
Status & Roadmap
- Done: event schema, append-only PostgreSQL store with insert-time hash chaining, chain rescan verifier, Docker Compose baseline.
- In progress: Merkle tree and Ed25519 tree-head signer, inclusion proof endpoint, first OPA policy (two-person approval) on the write path.
- Next: consistency proofs and gossip endpoint, seeded scenario simulator, replay CLI, then the audit dashboard.
Designed vs Built vs Measured
| Subsystem | Designed | Built | Measured |
|---|---|---|---|
| Event schema | Yes | Yes | Not applicable |
| Append-only PostgreSQL store with hash chaining | Yes | Yes | Not applicable |
| Chain rescan verifier | Yes | Yes | No verification benchmark yet |
| Docker Compose baseline | Yes | Yes | Not applicable |
| Merkle tree + Ed25519 tree-head signer | Yes | In progress | No measured results yet |
| Inclusion proof endpoint | Yes | In progress | No measured results yet |
| OPA policy gate (two-person approval) | Yes | In progress | No measured results yet |
| Consistency proofs + gossip endpoint | Yes | No | No measured results yet |
| Scenario simulator and replay CLI | Yes | No | No measured results yet |
| Audit dashboard | Yes | No | No measured results yet |
The schema, the append-only store, the hash-chain verifier, and the Compose baseline exist. Merkle proofs, the signer, the OPA gate, replay, and the dashboard are incomplete, and no benchmark has been run, so verification claims above are designs and protocols.
How I Built It
A versioned JSON schema for three event types (observation, recommendation, decision), stored in PostgreSQL with a writer role that physically cannot run UPDATE, DELETE, or TRUNCATE. Inserts compute the chain hash in a trigger-checked transaction. Deletion was tested to fail, not assumed to fail.
A verifier that recomputes every hash link across the table. Any edit to history invalidates every subsequent link, so tampering shows up as a broken chain, not a missing row.
A Go service folds pending entries into an RFC 6962-style Merkle tree every N seconds, signs the tree head with an Ed25519 key held behind a software seal, and publishes signed heads to a gossip-style endpoint so forks are detectable.
Rego policies evaluate every decision event before it is committed. The first policy is two-person approval: a simulated deconfliction approval needs two distinct operator identities within a time window. The verdict (allow or deny, plus the policy version) is logged alongside the event.
A seeded Python generator produces fictional aircraft tracks, a rule-based stub issues recommendations (re-route, hold, handoff), and scripted operators approve or override. Deterministic seeds mean any reviewer can reproduce an incident exactly.
A CLI (replay --entry N) folds events 1 through N through the state machine and renders the picture at that moment; a small web UI shows the log, proof status, and policy verdicts per decision.
What You Need
| Category | Item | Notes |
|---|---|---|
| Hardware | Any laptop or desktop | Everything runs in Docker Compose on the homelab |
| Software | Go, PostgreSQL, Open Policy Agent, Python, Docker Compose | Ed25519 via Go x/crypto; RFC 6962 describes the Merkle proof design this follows |
| Skills | Go or Python, SQL, basic Rego for OPA policies, comfort reading an RFC | The Merkle tree is the steepest part; Trillian's design documentation is the best study material |
| Accounts | None | No external services involved; the scenario data is generated locally |
Challenges and Lessons Learned
- Append-only has to be real. The database role enforces it; convention does not. (One honest caveat: grants bind the writer role, not the table owner or a superuser, so ownership stays with a separate admin role.) The self-attack tests (edit a middle row, delete rows, fork the tree) must all be caught by verification, or the design is decoration. I wrote those tests before trusting the chain.
- Hashes make tampering detectable, not impossible. The protection comes from independent parties comparing signed tree heads, which is why the gossip endpoint matters more than the hash function. A verifier nobody runs is decoration.
- Policy decisions are evidence too. Logging the OPA verdict and the policy version with every event keeps policy changes visible in history, so a rule change cannot hide inside a quiet deploy. Policy-as-code is only auditable if the code version is in the log.
- Replay is a fold, not a query. State at entry N is a pure function of events 1 through N. That single property is what makes after-action review reproducible, and it forced the state machine to stay deterministic.
- Inclusion and consistency proofs answer different questions. Inclusion proves an entry is in the log; consistency proves the log only ever grew. An auditor needs both, which is why the roadmap has both endpoints.
What Is Next
- Consistency proofs and the gossip endpoint.
- Seeded scenario simulator and the replay CLI.
- Audit dashboard with per-decision proof status and policy verdicts.
- Verification benchmarks (proof generation and verification cost at log scale) as measured results. Until then, verification claims on this page are designs and protocols.