80 lines
3.6 KiB
Markdown
80 lines
3.6 KiB
Markdown
|
# `pve-lab`: Local Proxmox VE test cluster (nested in libvirt/KVM)
|
||
|
|
|
||
|
|
Stands up a disposable **2-node Proxmox VE 9.2 cluster** plus a **dedicated
|
||
|
|
storage VM** that serves **both NFS and iSCSI**, all as nested VMs on this
|
||
|
|
machine via `qemu:///system`. Intended for testing software that drives a
|
||
|
|
Proxmox cluster.
|
||
|
|
|
||
|
|
```
|
||
|
|
pve-net (NAT 10.10.10.0/24, gw .1)
|
||
|
|
├── pve1 10.10.10.11 PVE node 4 vCPU / 8 GB / 48 GB
|
||
|
|
├── pve2 10.10.10.12 PVE node 4 vCPU / 8 GB / 48 GB
|
||
|
|
└── pvestore 10.10.10.13 Debian 13 2 vCPU / 2 GB (NFS + iSCSI target)
|
||
|
|
```
|
||
|
|
|
||
|
|
Shared storage exposed to the cluster:
|
||
|
|
- **lab-nfs** — NFS share `/srv/nfs` (images, ISOs, backups, containers)
|
||
|
|
- **lab-lvm** — shared LVM volume group `pve_shared` on an iSCSI LUN
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
- Linux host with KVM + **nested virtualization** enabled.
|
||
|
|
- `libvirt`, `virt-install`, `qemu-img`, `genisoimage`, `docker`, `sshpass`.
|
||
|
|
- `sudo` (the `0*`/`teardown` scripts manage the system libvirt instance).
|
||
|
|
- Internet access to `*.proxmox.com` and `cloud.debian.org`.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
Run in order from this directory:
|
||
|
|
```bash
|
||
|
|
./00-prereqs.sh # libvirtd, KVM checks, SSH key, dirs
|
||
|
|
./01-network.sh # define + start the pve-net network
|
||
|
|
./02-download-iso.sh # fetch + verify the Proxmox ISO
|
||
|
|
./03-build-autoinstall-isos.sh # bake per-node unattended-install ISOs (docker)
|
||
|
|
./04-install-nodes.sh # unattended-install pve1 + pve2 (several minutes)
|
||
|
|
./05-storage-vm.sh # provision the NFS + iSCSI storage VM
|
||
|
|
./06-form-cluster.sh # create the cluster + join pve2
|
||
|
|
./07-add-shared-storage.sh # wire lab-nfs + lab-lvm into the datacenter
|
||
|
|
./verify.sh # end-to-end checks incl. a cross-node migration
|
||
|
|
```
|
||
|
|
|
||
|
|
Config (IPs, sizing, password, ISO version) lives in [`lab.env`](lab.env).
|
||
|
|
|
||
|
|
## Access
|
||
|
|
- Web UI: <https://10.10.10.11:8006> and <https://10.10.10.12:8006>
|
||
|
|
- Login: `root` / value of `ROOT_PASSWORD` in `lab.env`
|
||
|
|
- SSH: `ssh -i .ssh/id_lab root@10.10.10.11` (key auto-generated by `00`)
|
||
|
|
|
||
|
|
## Reset / rebuild
|
||
|
|
```bash
|
||
|
|
./teardown.sh # remove VMs, network, images, build artifacts
|
||
|
|
./teardown.sh --keep-iso # keep the downloaded ISOs for a faster rebuild
|
||
|
|
```
|
||
|
|
|
||
|
|
## Notes
|
||
|
|
- The PVE nodes power off after install (`reboot-mode = "power-off"`) so
|
||
|
|
`virt-install` cleanly hands off to disk-boot; they are then started normally.
|
||
|
|
- `proxmox-auto-install-assistant` isn't packaged for Fedora, so `03` runs it
|
||
|
|
inside a `debian:trixie` container against the Proxmox `pve-no-subscription`
|
||
|
|
repo. Docker only needs the user to be in the `docker` group (no sudo).
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
- **Storage VM can't install packages / no internet in guests.** On firewalld
|
||
|
|
hosts, libvirt's NAT network frequently lacks FORWARD-accept rules, so guests
|
||
|
|
reach the host gateway (`10.10.10.1`) but nothing beyond. `01-network.sh` now
|
||
|
|
adds explicit `iptables` forward + masquerade rules for the subnet to fix this;
|
||
|
|
`teardown.sh` removes them. The storage VM's apt is also forced to IPv4
|
||
|
|
(mirrors resolve to unreachable IPv6 on this NAT). Verify from the host:
|
||
|
|
`ssh -i .ssh/id_lab root@10.10.10.13 ping -c2 1.1.1.1`.
|
||
|
|
- **Never run `virsh net-destroy` on a network with running VMs.** It tears down
|
||
|
|
the bridge and orphans the VMs' tap interfaces; you must then restart the
|
||
|
|
domains (`virsh destroy <dom> && virsh start <dom>`) to re-attach them.
|
||
|
|
|
||
|
|
## License & genAI
|
||
|
|
|
||
|
|
This project is licensed under the terms of the MIT license. See `LICENSE` for
|
||
|
|
details.
|
||
|
|
|
||
|
|
I believe it is important to disclose when generative AI ("genAI") was used in
|
||
|
|
the creation of a work, as is the case here. Much of this was authored using
|
||
|
|
Claude Sonnet 5, though I have reviewed and tested by hand.
|
||
|
|
|