# `pve-lab`: Local Proxmox VE test cluster (nested in libvirt/KVM)

Stands up a disposable **2-node Proxmox VE 9.2 cluster** plus a **dedicated
storage VM** that serves **both NFS and iSCSI**, all as nested VMs on this
machine via `qemu:///system`. Intended for testing software that drives a
Proxmox cluster.

```
pve-net  (NAT 10.10.10.0/24, gw .1)
├── pve1      10.10.10.11   PVE node   4 vCPU / 8 GB / 48 GB
├── pve2      10.10.10.12   PVE node   4 vCPU / 8 GB / 48 GB
└── pvestore  10.10.10.13   Debian 13  2 vCPU / 2 GB  (NFS + iSCSI target)
```

Shared storage exposed to the cluster:
- **lab-nfs** — NFS share `/srv/nfs` (images, ISOs, backups, containers)
- **lab-lvm** — shared LVM volume group `pve_shared` on an iSCSI LUN

## Requirements
- Linux host with KVM + **nested virtualization** enabled.
- `libvirt`, `virt-install`, `qemu-img`, `genisoimage`, `docker`, `sshpass`.
- `sudo` (the `0*`/`teardown` scripts manage the system libvirt instance).
- Internet access to `*.proxmox.com` and `cloud.debian.org`.

## Usage
Run in order from this directory:
```bash
./00-prereqs.sh                 # libvirtd, KVM checks, SSH key, dirs
./01-network.sh                 # define + start the pve-net network
./02-download-iso.sh            # fetch + verify the Proxmox ISO
./03-build-autoinstall-isos.sh  # bake per-node unattended-install ISOs (docker)
./04-install-nodes.sh           # unattended-install pve1 + pve2 (several minutes)
./05-storage-vm.sh              # provision the NFS + iSCSI storage VM
./06-form-cluster.sh            # create the cluster + join pve2
./07-add-shared-storage.sh      # wire lab-nfs + lab-lvm into the datacenter
./verify.sh                     # end-to-end checks incl. a cross-node migration
```

Config (IPs, sizing, password, ISO version) lives in [`lab.env`](lab.env).

## Access
- Web UI: <https://10.10.10.11:8006> and <https://10.10.10.12:8006>
- Login: `root` / value of `ROOT_PASSWORD` in `lab.env`
- SSH: `ssh -i .ssh/id_lab root@10.10.10.11` (key auto-generated by `00`)

## Reset / rebuild
```bash
./teardown.sh             # remove VMs, network, images, build artifacts
./teardown.sh --keep-iso  # keep the downloaded ISOs for a faster rebuild
```

## Notes
- The PVE nodes power off after install (`reboot-mode = "power-off"`) so
  `virt-install` cleanly hands off to disk-boot; they are then started normally.
- `proxmox-auto-install-assistant` isn't packaged for Fedora, so `03` runs it
  inside a `debian:trixie` container against the Proxmox `pve-no-subscription`
  repo. Docker only needs the user to be in the `docker` group (no sudo).

## Troubleshooting
- **Storage VM can't install packages / no internet in guests.** On firewalld
  hosts, libvirt's NAT network frequently lacks FORWARD-accept rules, so guests
  reach the host gateway (`10.10.10.1`) but nothing beyond. `01-network.sh` now
  adds explicit `iptables` forward + masquerade rules for the subnet to fix this;
  `teardown.sh` removes them. The storage VM's apt is also forced to IPv4
  (mirrors resolve to unreachable IPv6 on this NAT). Verify from the host:
  `ssh -i .ssh/id_lab root@10.10.10.13 ping -c2 1.1.1.1`.
- **Never run `virsh net-destroy` on a network with running VMs.** It tears down
  the bridge and orphans the VMs' tap interfaces; you must then restart the
  domains (`virsh destroy <dom> && virsh start <dom>`) to re-attach them.

## License & genAI

This project is licensed under the terms of the MIT license. See `LICENSE` for
details.

I believe it is important to disclose when generative AI ("genAI") was used in
the creation of a work, as is the case here. Much of this was authored using
Claude Sonnet 5, though I have reviewed and tested by hand.