P5x is a set of tools for running a Kubernetes cluster on LXC containers in Proxmox. The `csi-driver` component provides a Kubernetes `StorageClass` and provisioner that will automatically create, migrate, and mount Proxmox volumes on a designated network storage for use with Kubernetes `PersistentVolumes`.
Why? This allows the storage backend to be configured in Proxmox, instead of requiring a separate Kubernetes storage solution or local provisioner. It also makes Kubernetes volumes appear as native Proxmox disks _separate_ from the storage of the LXC container they're running on, which is much closer to a traditional cloud environment like EKS/GKE.
> **Before setting up the `csi-driver`, you must first deploy the P5x `api-server` component.** The `api-server` is required for all P5x components and manages communication between Kubernetes and the underlying Proxmox infrastructure.
Once you have deployed the `api-server`, deploy the Kubernetes resources for the `csi-driver`:
```shell
kubectl apply -f deploy
```
In your Kubernetes cluster, in the `p5x-system` namespace, you should now see a `p5x-csi-controller-0` pod along with all the `p5x-csi-node-*` pods from the CSI DaemonSet.
### Verify API Connectivity & Get SSH Pubkey
Run the following command to verify that the CSI controller can communicate with the API server. This prints out P5x's SSH public key. Note this down, as we'll use it in the next step.
Prerequisite: The Kubernetes nodes (LXC containers) must have an SSH server running with key-based `root` authentication enabled. Example for EL-based Linux distros:
```shell
dnf install -y openssh-server
systemctl enable --now sshd
```
On each of the Kubernetes nodes (LXC containers) that P5x will interact with (i.e. create disks on, mount disks to), add the public key from the previous step to the `/root/.ssh/authorized_keys` file.
### Registering Nodes
I am still working on a system for the P5x API server to auto-discover K8s nodes (LXC containers) or receive them from a config file. For now, you can run the following command to manually register a node with P5x:
-X POST --location "http://api.p5x-system.svc.cluster.local:3450/node" \
-H "Content-Type: application/json" \
-d '{
"hostname": "k8s-node-hostname",
"pve_id": 110,
"pve_host": "lxc-container-hostname",
"assigned_ip": "lxc-container-ip-address",
"assigned_subnet": 24,
"is_permanent": true
}'
```
Parameters:
-`hostname` - The hostname of the K8s node as it appears in the Kubernetes cluster (i.e. `kubectl get nodes`)
-`pve_id` - The Proxmox ID of the LXC container
-`pve_host` - The hostname of the PVE node where the LXC container resides as it appears in the Proxmox VE web interface
-`assigned_ip` - Resolvable LAN IP address of the LXC container - used for SSH access
-`assigned_subnet` - Integer subnet identifier for the LAN IP of the LXC container
-`is_permanent` - For future use - this should be true for manually-provisioned nodes
You can run this command once for each node in your K8s cluster. After doing so, P5x will be able to manage volumes for pods on those nodes.
## Usage
Once the CSI driver is set up and the K8s nodes are registered with the API server, you can use the P5x `StorageClass` for storage in Kubernetes. Here's an example `PersistentVolumeClaim`:
```yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: example-pvc-1
namespace: my-namespace
spec:
storageClassName: p5x
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
```
Once the PVC is created and mounted to a pod, the CSI driver will create a new Proxmox volume, migrate it to the correct LXC container, and mount it into the K8s pod for use.
For a complete example, see the `examplepod.yaml` file, but you should see something like:
Deleting the pod will unmount the volume(s), but will not delete them. Deleting the PVCs will permanently delete the disks.
## Q&A
- Why does P5x need `root` access to Proxmox and my K8s LXC containers?
- The fundamental answer to both of these questions is "so it can dynamically unmount/migrate volumes without rebooting the LXC containers."
- You cannot dynamically _unmount_ a Proxmox volume from an LXC container using the Proxmox API, so in order to do so without requiring a reboot, P5x uses its SSH access to the LXC container to `unmount` the disk from the LXC container's filesystem.
- Okay, but why Proxmox?
- Reason 1: Some of these disk operations cannot be done via the Proxmox API and require access to read/modify the `/etc/pve/lxc/*.conf` config for the LXC container directly.
- Reason 2: For security reasons, when Proxmox mounts a disk in an LXC container, it also shadow mounts the disk in an isolated namespace on the Proxmox host itself. This prevents certain vulnerabilities where a hostile LXC container (ab)uses `unmount` internally to gain access to the host filesystem. However, it also prevents dynamically unmounting a volume from the LXC container without rebooting the container. P5x uses its SSH access to the Proxmox node to `unmount` the disk from the shadow mount in the isolated namespace.
- What are the `carrier-...` LXC containers in my Proxmox VE cluster?
- Currently, Proxmox does not support dynamically migrating an "Unused Disk" from an LXC container on one physical PVE host to a different PVE host.
- To accomplish this, P5x provisions a tiny dummy LXC container on the same PVE host, moves the disk over to that dummy container, migrates the entire dummy container to the new PVE host (which _is_ supported), then moves the disk off of the dummy container.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.
