Resilient Server Infrastructure Build
You will design and build resilient server infrastructure: virtualization, flexible storage, managed services, backups, and monitoring. By the end you will have a documented estate that recovers cleanly from failure.
Learning Objectives
- Provision VMs with flexible LVM-backed storage.
- Run services under systemd with auto-restart.
- Add tested backups and monitoring.
- Time: ~16 hours · Difficulty: Advanced · Prereqs: the Linux server labs.
Architecture Overview
graph TD
HV[Hypervisor] --> VMs[Service VMs]
VMs --> LVM[LVM storage]
VMs --> SD[systemd services]
VMs --> BK[restic backups]
VMs --> Mon[Prometheus]
Environment Setup
- A KVM/Proxmox host with spare disks.
- restic and a backup target; Prometheus + Grafana.
Step-by-Step Execution
01
Provision VMs on LVM storage
lvcreate -L 30G -n vm-disk vg0 && virt-install --name app01 --disk path=/dev/vg0/vm-disk --import02
Run services with auto-restart and back them up
systemctl enable --now app.service && restic -r /backup/repo backup /srv03
Add monitoring
$ curl -s localhost:9090/api/v1/targets | grep -o '"health":"up"' | wc -l
4
Progress So Far
graph LR
A[01 VMs + LVM] -->|done| B[02 Services + backups]
B -->|done| C[03 Monitoring]
style A fill:#1a4a1a,stroke:#00ff00,color:#fff
style B fill:#1a4a1a,stroke:#00ff00,color:#fff
style C fill:#1a4a1a,stroke:#00ff00,color:#fff
Testing & Validation
restic -r /backup/repo restore latest --target /tmp/restore --include /srv/config && systemctl is-active app.serviceA restore should succeed and the service should be active. Kill the service and confirm systemd restarts it. The estate is resilient and recoverable.
Troubleshooting
- VM won't boot from LVM: confirm the disk path and that the LV is not in use.
- Backup untested: schedule periodic restore drills, not just backups.
- Targets down in Prometheus: verify exporters are running and reachable.
Extension Ideas
- Add HA/clustering as in Proxmox Zero-Trust.
- Centralize logs and alerts.
- Promote to the On-Prem Capstone.
Key Results
- Provisioned service VMs on flexible LVM storage.
- Ran services with systemd auto-restart.
- Verified backups with a real restore.
- Monitored the estate with healthy Prometheus targets.