>
Taki Sadik
Cybersecurity & IT Professional, North York, ON, Canada
Contact

Nested Virtual Machines with KVM, libvirt, and virsh

AdvancedLinux~4 hours
KVMlibvirtvirt-installvirsh

I enabled nested virtualization on KVM and provisioned guest VMs inside a VM with libvirt, virt-install, and virsh. Nesting let me build multi-node clustering and hypervisor labs on a single physical host with no additional hardware.

Objective & Context

Nested KVM exposes the CPU virtualization extensions to a guest so it can itself run VMs. This lab turns on nesting, scripts guest creation with virt-install, and manages lifecycle with virsh, the technique behind the Proxmox and Kubernetes cluster labs.

Learning Objectives

  • Verify CPU virtualization extensions and enable KVM nesting.
  • Persist the nested=1 module option across reboots.
  • Provision inner guests non-interactively with virt-install.
  • Manage guest lifecycle (start, shutdown, snapshot) with virsh.
  • Explain the performance and use-case trade-offs of nesting.

Environment & Prerequisites

Prerequisites

  • A Linux host with KVM, libvirt, and CPU virtualization extensions (or a cloud VM with nesting allowed).
  • Sufficient RAM and CPU for nested guests (2 vCPU and 2 GB per inner guest minimum).
  • A cloud image or ISO for the inner guest.
  • Root or sudo access and membership in the libvirt group for management.

Step-by-Step Execution

1. Enable nested virtualization [ROOT REQUIRED]

sudo modprobe -r kvm_intel; sudo modprobe kvm_intel nested=1; cat /sys/module/kvm_intel/parameters/nested

Expected output:

Y

What this teaches: kvm_intel nested=1 exposes VMX to guests so the L1 guest can run its own hypervisor; reading the parameter back confirms the module accepted it (Y means enabled, N means disabled). The kernel has enabled nesting by default since v4.20, so on a modern host this often already reads Y and the reload is a confirmation. To persist across reboots regardless of defaults, write options kvm_intel nested=1 to /etc/modprobe.d/kvm-nested.conf with echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf; on AMD hosts the module is kvm_amd. The reload fails with "module in use" while guests run, so shut them down first.

2. Provision an inner guest with virt-install

sudo virt-install --name inner1 --memory 2048 --vcpus 2 --disk size=10 --osinfo debian12 --cdrom /var/lib/libvirt/images/debian-12-amd64-netinst.iso --noautoconsole

Expected output:

Starting install...
Creating domain...  |  10.0 GB
Domain creation completed.

What this teaches: virt-install turns guest creation into one scripted command: name, memory, vCPUs, disk, and OS profile (--osinfo) pick the right virtual hardware automatically. Note the install media choice: --cdrom boots the installer for a fresh OS install, while --import is only valid when --disk points at an already installed disk image (it skips the installer and builds the guest around it). Using --import together with --disk size=10 and no image path fails, so pick the flag that matches your media: an ISO gets --cdrom, a downloaded cloud image gets --disk /path/to/image.qcow2 --import. sudo is needed for the default qemu:///system connection; members of the libvirt group can drop it.

3. Manage with virsh

virsh list --all && virsh start inner1

Expected output:

 Id   Name     State
--------------------------
 3    inner1   running

Expected output:

Domain 'inner1' started

What this teaches: virsh list --all shows every defined guest and state; virsh start boots by name. virsh is the CLI for the full libvirt API: console, snapshot, migrate, destroy.

How It Works

Nested virtualization works by passing the CPU's virtualization extensions through to the guest. Normally the hypervisor hides VMX/SVM from guests; with nesting enabled, the L1 guest sees the extensions and can run KVM itself, creating L2 guests. Each level adds trap-and-emulate overhead, so nested guests run slower, but for lab and CI purposes the convenience outweighs the cost.

libvirt is the management layer that turns hypervisor APIs into a stable interface: virsh commands, XML domain definitions, and storage/network pools. virt-install is a thin provisioner on top that generates the XML from flags, which is why scripted lab environments prefer it over clicking through virt-manager.

The canonical use case is hypervisor development and training: the Proxmox and Kubernetes labs in the work section rely on nested guests to simulate multi-node clusters on one physical host. Understanding the nesting toggle is what makes those environments reproducible anywhere.

Validation & Testing

Confirm nested reads Y, then boot an inner guest and verify it sees virtualization extensions (egrep -c '(vmx|svm)' /proc/cpuinfo > 0). Pass criteria: nesting enabled, inner guest boots, and it can itself run a container or VM.

Troubleshooting

SymptomLikely causeFix
modprobe: module in useGuests are running, so kvm_intel cannot reloadShut down all guests first, or set the option via /etc/modprobe.d and reboot
Inner guest reports KVM not availableNesting did not apply to the L1 guestConfirm Y in /sys/module/kvm_intel/parameters/nested on the host
virt-install: --osinfo unknownosinfo-db is outdated or missingUpdate osinfo-db or use --os-variant with a close match
virsh: permission deniedUser is not in the libvirt groupAdd with usermod -aG libvirt $USER and log back in
Nested guest is painfully slowExpected: double trap-and-emulate overheadReduce L2 vCPUs, enable host CPU passthrough, accept it for labs only
Cloud provider blocks nestingThe provider hides virtualization extensionsUse a provider/instance type with nested virtualization support
Advanced: Troubleshooting
  • nested = N: the module must be reloaded with nested=1; persist via modprobe.d.
  • Inner guest no VT: set the L1 VM CPU mode to host-passthrough.
  • Poor performance: nesting adds overhead; allocate generous host resources.

Key Results

  • Enabled nested KVM and booted guests inside a VM.
  • Built multi-node lab topologies on a single physical host.
  • Scripted reproducible guest creation with virt-install.
  • Managed full VM lifecycle through virsh.

Key Takeaways

  • nested=1 exposes VMX/SVM to guests, enabling L2 VMs.
  • Persist module options in /etc/modprobe.d, not just modprobe.
  • virt-install scripts guest creation; virsh manages lifecycle.
  • Nesting costs performance; it is a lab tool, not a production pattern.
  • libvirt group membership controls who can manage guests.

Practice Challenges

Challenge 1: Two-level lab

Goal: on the L1 guest, install KVM and boot an L2 guest, then prove nesting with lscpu inside L2.

Hint: repeat the host steps inside L1 after confirming the flag.

Challenge 2: Snapshot round-trip

Goal: snapshot inner1, break its network config, then revert and confirm connectivity returns.

Hint: virsh snapshot-create-as and snapshot-revert.