Kubernetes Infrastructure

Overview

This project is a self-designed, self-built Kubernetes homelab running across 3 desktop PCs and 4 laptops, interconnected through a single managed switch, a dedicated router handling inter-VLAN routing and NAT, and a wireless radio providing mesh backhaul for the laptop nodes. The goal was to replicate a production-grade container orchestration environment entirely from consumer hardware — no cloud, no rented rack space, no managed services.

The cluster runs kubeadm-bootstrapped Kubernetes on Ubuntu Server VMs provisioned inside Proxmox, with Calico as the CNI for pod networking and network policy enforcement, MetalLB for bare-metal LoadBalancer services, Longhorn for distributed persistent storage across nodes, and a full observability stack built on Prometheus + Grafana + Loki. Deployments are automated through Bash scripts and Docker Compose for auxiliary services, with GitOps workflows managing application manifests.

Every component — from the physical cable runs and VLAN trunking to the Kubernetes RBAC policies and Grafana dashboards — was planned, configured, and documented by hand. The cluster currently hosts all self-hosted services (Nextcloud, AI model, DNS, email, wiki, search engine, photo server) in production.

Physical Topology

The entire infrastructure runs from a single physical location. Three desktop PCs serve as the primary compute and storage nodes, while four laptops act as lightweight worker nodes and provide redundancy. All wired nodes connect to a managed switch with 802.1Q VLAN trunking. The router handles inter-VLAN routing, DHCP reservation, NAT to the public internet, and port forwarding for externally-facing services. A wireless radio bridges the laptop nodes into the cluster network over a dedicated 5 GHz backhaul link.

Network Architecture

Network segmentation is enforced at the switch level using three VLANs. The router performs inter-VLAN routing with firewall rules restricting lateral movement between segments. This mirrors enterprise network design where management, application, and storage traffic are isolated to prevent blast radius expansion during a compromise.

Kubernetes Architecture

Tech Stack

Build Process

# Switch VLAN configuration (CLI example)
enable
configure terminal

vlan 10
  name MANAGEMENT
vlan 20
  name CLUSTER
vlan 30
  name STORAGE

# Trunk port to router
interface GigabitEthernet0/1
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30

# Access ports for desktop nodes (VLAN 20)
interface range GigabitEthernet0/2-4
  switchport mode access
  switchport access vlan 20

# Trunk port to wireless radio
interface GigabitEthernet0/5
  switchport mode trunk
  switchport trunk allowed vlan 20

# Create a cloud-init template VM on Proxmox
qm create 9000 --name ubuntu-cloud --memory 4096 --cores 2 \
  --net0 virtio,bridge=vmbr0,tag=20 \
  --net1 virtio,bridge=vmbr0,tag=30 \
  --scsihw virtio-scsi-single

# Import Ubuntu cloud image
qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0

# Configure cloud-init defaults
qm set 9000 --ciuser taki --cipassword changeme \
  --sshkeys ~/.ssh/authorized_keys \
  --ipconfig0 ip=dhcp \
  --ipconfig1 ip=10.30.30.X/24,gw=10.30.30.1

# Clone template for each node
for i in 1 2 3 4 5 6; do
  qm clone 9000 10${i} --name k8s-node-${i} --full
  qm start 10${i}
done

# Run on ALL nodes
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

sudo swapoff -a && sudo sed -i '/swap/d' /etc/fstab

# Install containerd
sudo apt install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

# Install kubeadm, kubelet, kubectl
sudo apt install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update && sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Initialize control plane (PC-01)
sudo kubeadm init \
  --pod-network-cidr=192.168.0.0/16 \
  --apiserver-advertise-address=10.20.20.10 \
  --control-plane-endpoint=10.20.20.10:6443

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config

# Join workers (run on each worker node)
sudo kubeadm join 10.20.20.10:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

# Install Calico operator and custom resources
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/custom-resources.yaml

# Verify Calico pods are running
kubectl get pods -n calico-system -w

# Default-deny ingress NetworkPolicy (applied per namespace)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress

# Allow only Nginx Ingress to reach web pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-web
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

# Install MetalLB
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml

# Configure IP address pool
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: cluster-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.20.20.200-10.20.20.250

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: cluster-l2
  namespace: metallb-system

# Install Nginx Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/baremetal/deploy.yaml

# Install cert-manager for automated TLS
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml

# ClusterIssuer for Let's Encrypt
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx

# Install Longhorn
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.2/deploy/longhorn.yaml

# Set as default StorageClass
kubectl patch storageclass longhorn -p \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# Verify volumes and nodes
kubectl -n longhorn-system get pods
kubectl -n longhorn-system get nodes.longhorn.io

# Example PVC using Longhorn
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nextcloud-data
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 100Gi

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Add repos
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install kube-prometheus-stack
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword='secureDashboard' \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=longhorn \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

# Install Loki + Promtail
helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set promtail.enabled=true \
  --set loki.persistence.enabled=true \
  --set loki.persistence.storageClassName=longhorn \
  --set loki.persistence.size=20Gi

# Expose Grafana via Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [grafana.tyfsadik.org]
      secretName: grafana-tls
  rules:
    - host: grafana.tyfsadik.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: monitoring-grafana
                port:
                  number: 80

Services Running on the Cluster

Deployment Workflow

Challenges & Solutions

• Laptop nodes dropping from cluster over Wi-Fi: The laptop worker nodes periodically lost connectivity over the wireless bridge, causing kubelet to miss heartbeats and the control plane to mark them as NotReady. Resolved by deploying a dedicated 5 GHz radio in bridge mode with a fixed channel, disabling power management on the laptop NICs (iw dev wlan0 set power_save off), and increasing the node-monitor-grace-period to 60s to tolerate brief interruptions without evicting pods.
• Longhorn replication saturating the network: Initial deployment placed storage replication on the same VLAN as cluster traffic, causing API server latency spikes during large write operations. Resolved by creating a dedicated VLAN 30 for storage traffic and configuring Longhorn to use the VLAN 30 interface for replication.
• MetalLB ARP conflicts with router DHCP: MetalLB’s L2 mode ARP responses conflicted with the router’s DHCP leases when the IP pool overlapped with the DHCP range. Fixed by reserving 10.20.20.200–250 as a static range excluded from DHCP and configuring MetalLB to only advertise within that range.
• etcd performance on a single control plane node: With only one control plane node, etcd write latency spiked during heavy scheduling. Mitigated by placing etcd data on the NVMe drive (not SSD), tuning heartbeat-interval and election-timeout, and ensuring no other I/O-heavy workloads run on PC-01.
• TLS certificate provisioning for multiple subdomains: cert-manager’s HTTP-01 solver required each subdomain to be publicly reachable, which conflicted with internal-only services. Resolved by using DNS-01 challenge validation for internal services and HTTP-01 for public-facing ones.
• Resource contention on 8 GB laptop nodes: Laptops with only 8 GB RAM struggled when Longhorn and Prometheus exporters consumed too much memory alongside application pods. Fixed by applying resource limits and requests to all pods, tainting the laptop nodes with node-role=lightweight:PreferNoSchedule, and using node affinity rules to keep heavy workloads on the desktop nodes.

Security Hardening

• RBAC: Least-privilege service accounts per namespace; no pods run with cluster-admin
• NetworkPolicy: Default-deny ingress on all namespaces; explicit allow rules per service
• Pod Security Standards: restricted profile enforced via Pod Security Admission; no privileged containers
• Image Scanning: Trivy scans on all container images before deployment; CVE alerts sent to Grafana
• Secrets Management: Kubernetes Secrets encrypted at rest with aescbc encryption provider
• SSH Hardening: Key-only authentication, fail2ban on all nodes, management access restricted to VLAN 10
• Firewall Rules: Router ACLs restrict inter-VLAN traffic; only necessary ports open between segments

What I Learned

• End-to-end Kubernetes cluster lifecycle: bootstrapping, upgrading, scaling, and troubleshooting with kubeadm
• Bare-metal networking for Kubernetes: VLAN segmentation, BGP with Calico, ARP-based load balancing with MetalLB
• Distributed storage engineering: Longhorn replication, IOPS tuning, and failure recovery across physical nodes
• Enterprise-grade observability: Prometheus metric design, Grafana alerting pipelines, Loki log correlation
• Physical infrastructure design: cable management, switch VLAN configuration, wireless backhaul for cluster nodes
• Security-first architecture: network micro-segmentation, RBAC design, pod security standards, image vulnerability scanning
• Resource management on heterogeneous hardware: taints, tolerations, affinity rules, and resource quotas to balance workloads across nodes with different capabilities
• The discipline of documentation: every configuration change logged, every decision recorded, every diagram kept current