TYF-AI — Self-Hosted Local AI Security Platform
A hardened, fully local AI inference stack: llama.cpp with CUDA, Open WebUI behind authenticated reverse proxy, and per-model sandboxing on Proxmox with GPU passthrough.
Overview
TYF-AI removes cloud AI dependency entirely by serving quantized GGUF models on local hardware while treating the inference engine as an untrusted workload. Open WebUI sits behind a Caddy reverse proxy that enforces authentication and API-key rotation, and each model process runs inside a Firejail sandbox to contain a compromised runtime. The result is a private assistant that served 50+ concurrent queries at sub-2-second latency without a single external API call.
System Architecture
flowchart TB
U[Browser] -->|HTTPS| C[Caddy Reverse Proxy
auth + rate limit] C --> W[Open WebUI] W -->|REST| LC[llama.cpp CUDA server] LC --- GG[(GGUF Q4_K_M / Q5_K_M)] LC -. confined by .-> FJ[Firejail Sandbox] subgraph PVE["Proxmox VE 8.1"] VM[AI VM] -->|PCI passthrough| GPU[NVIDIA GPU] end LC --- VM
auth + rate limit] C --> W[Open WebUI] W -->|REST| LC[llama.cpp CUDA server] LC --- GG[(GGUF Q4_K_M / Q5_K_M)] LC -. confined by .-> FJ[Firejail Sandbox] subgraph PVE["Proxmox VE 8.1"] VM[AI VM] -->|PCI passthrough| GPU[NVIDIA GPU] end LC --- VM
Threat Model: Local vs Cloud AI (STRIDE)
| STRIDE | Cloud AI | TYF-AI (Local) |
|---|---|---|
| Spoofing | Provider account takeover | Caddy auth + rotating API keys |
| Tampering | Opaque model updates | Pinned GGUF checksums |
| Information Disclosure | Prompts leave your network | Data never leaves the host |
| Denial of Service | Vendor rate limits | Local rate limiting in Caddy |
| Elevation of Privilege | N/A | Firejail confines model process |
sequenceDiagram
participant U as User
participant C as Caddy
participant W as Open WebUI
participant L as llama.cpp (Firejail)
U->>C: HTTPS + API key
C->>C: validate key + rate limit
C->>W: forward
W->>L: completion request
L-->>W: streamed tokens
W-->>U: response
Deployment
1
GPU passthrough on Proxmox
Bind the GPU to vfio-pci so the AI VM owns it exclusively, enabling CUDA-accelerated inference.
echo "vfio-pci ids=10de:2231" > /etc/modprobe.d/vfio.conf && update-initramfs -u2
Run the sandboxed inference server
firejail --net=none --private ./llama-server -m models/model-q4_K_M.gguf --n-gpu-layers 35 --port 80803
Authenticated reverse proxy (Caddyfile)
ai.tyfsadik.org { reverse_proxy localhost:3000; rate_limit { zone ai { events 60 window 1m } } }Security Hardening
- Prompt injection: system-prompt isolation and output filtering on tool-use responses.
- Model integrity: SHA-256 pinning of GGUF files to detect the "Mind the Gap" GGUF tampering class.
- Data leakage:
--net=nonedenies the model process outbound network access. - Key management: rotating API keys issued per client, revocable at the proxy.
Key Results
- Served 50+ concurrent queries at under 2 seconds latency on local hardware.
- Reduced external AI API dependency by 100% for internal tooling.
- Isolated model execution, preventing a compromised runtime from reaching the host or network.
- Maintained 99.9% uptime alongside other self-hosted services on the same node.