# Homelab Provisioning Layer This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus. `./jeannie up` drives this layer in auto mode when the Pimox host is reachable at `192.168.100.80`, `qm` is installed, and the existing bridge `vmbr0` is present. The automation creates VM definitions only; it validates the bridge and refuses to edit Orange Pi host networking. ## What It Installs - `dnsmasq` for proxyDHCP and TFTP - `nginx` for preseed, installer, and guest-prep scripts - Debian 13 arm64 netboot assets under `/opt/homelab-provisioning` - a preseed file for unattended Debian install - a GRUB PXE rescue menu and HTTP rescue page at `/rescue/` - guest prep scripts that install Kubernetes tools, containerd, qemu guest agent, cloud-init, OpenEBS dependencies, cgroup prerequisites, swap disablement, and the local registry trust path - Kubernetes tools from the pinned minor in `TF_VAR_kubernetes_minor_version`, defaulting to `v1.36` - `containerd.io` from Docker's Debian repository so worker runtimes use the same package source as the Debian control-plane host - kernel boot options for cgroup support through `TF_VAR_kernel_cgroup_boot_options` - a template sealing script at `/usr/local/sbin/homelab-prepare-template.sh` inside the installed VM that verifies cgroup boot state before sealing ## Apply From Debian The normal path is: ```bash cd ~/my-homelab-configs ./jeannie up ``` For manual provisioning-only testing, find the LAN interface on the Debian server: ```bash ip -br addr ``` Apply the provisioning layer: ```bash cd ~/my-homelab-configs tofu -chdir=bootstrap/provisioning init TF_VAR_provisioning_interface=enp1s0 tofu -chdir=bootstrap/provisioning apply ``` Override `TF_VAR_provisioning_interface` with the interface that serves the Pimox VM network. The default VM user is `jv`. The account uses `/home/jv/.ssh/id_ed25519.pub` from the Debian server when that key exists, and the password is locked by default. Set `TF_VAR_template_user_ssh_authorized_keys` or `TF_VAR_template_user_password_hash` before applying if you want different access. Clones should get their intended hostname through cloud-init or the later clone automation. If they boot with the template hostname, the first-boot service generates a unique fallback name using `TF_VAR_clone_hostname_prefix`. ## Pimox VM Template Flow Create an arm64 VM in Pimox with UEFI firmware, a virtio disk, and a NIC on the same LAN as the Debian provisioning host. Put network boot first. PXE should load `grubaa64.efi` and display a GRUB menu. The default entry keeps the automated template flow intact: it boots the Debian installer, fetches the preseed from `http://192.168.100.73:8088/preseed/debian13-arm64-worker.cfg`, and installs the golden image. The same menu also includes rescue-oriented entries: - Debian 13 arm64 rescue shell - Debian 13 arm64 expert/manual install - Debian 13 arm64 automated install, verbose - UEFI firmware settings - Exit PXE and continue local boot - Reboot The Debian provisioning host also publishes a rescue landing page: ```bash curl http://192.168.100.73:8088/rescue/ ``` If your Pimox firmware needs a different Debian arm64 EFI loader, override `TF_VAR_pxe_boot_file`. After the first successful boot, run this inside the VM before converting it to a template: ```bash sudo /usr/local/sbin/homelab-prepare-template.sh sudo poweroff ``` Convert the powered-off VM to a Pimox template. Clone it for each new worker, set a unique hostname and IP address through cloud-init or DHCP reservation, then add it to `bootstrap/cluster/variables.tf` or a `.tfvars` file: ```hcl worker_nodes = { pimox01 = { host = "192.168.100.90" user = "jv" node_name = "pimox01" ssh_key_path = "/home/jv/.ssh/id_ed25519" } } ``` Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH. ## Pimox Automation `./jeannie up` runs Pimox provisioning by default and sets `TF_VAR_pimox_template_builder_enabled=true` when Pimox is reachable. The layer SSHes into the Pimox host, create the template-build VM with `qm`, boot it from PXE, wait for the installed VM over SSH, run `/usr/local/sbin/homelab-prepare-template.sh`, power it off, switch boot order back to disk first, and run `qm template`. The template sealing step discovers the installed VM IP through qemu-guest-agent, so a DHCP reservation is no longer required for the temporary template-build VM. If you still want to force a known address, set `TF_VAR_pimox_template_build_host`. Leave it unset for the normal autodiscovery path. Once a guest IP or explicit guest host is known, the seal step now uses a shorter SSH deadline of `TF_VAR_pimox_template_guest_ssh_timeout_seconds=300` by default so bad IP/SSH assumptions fail quickly instead of consuming the full template build timeout. ```bash ./jeannie up ``` Defaults match the observed Pimox template VM shape: OVMF firmware, virtio networking, virtio-scsi disk, `vmbr0`, `local` template storage, 1 socket with 2 cores, and 8 GiB memory. The native `qm` CPU topology is the only `-smp` argument; the provisioning stack removes legacy `args` values to prevent a second conflicting `-smp` option. Override `TF_VAR_pimox_template_scsi0`, `TF_VAR_pimox_template_efidisk0`, `TF_VAR_pimox_template_cores`, or `TF_VAR_pimox_template_memory` if the Orange Pi template layout changes. `./jeannie up` also creates or reuses worker clones after the template exists. It defaults to one worker, VMID `9010`, name `pimox-worker-01`, deterministic locally administered MAC address, 1 socket with 2 cores, 8 GiB RAM, `opi5_ssd` clone storage, and qemu-guest-agent IP discovery when that Pimox storage is available. New workers are full clones created with `qm clone --storage`, so the template can remain on `local` while worker disks land on the Orange Pi SSD-backed non-local Pimox datastore. Keep durable data on the HP Debian laptop's `data-vg` or external backups and treat Orange Pi VM disks as rebuildable worker capacity. Set `LAB_PIMOX_WORKER_REPLACE_EXISTING=true` to destroy and recreate existing worker VMs from the current template. The pipeline refuses `LAB_PIMOX_WORKER_STORAGE=local` so only the template VM lives on local storage. Worker networking defaults to static because the ISP modem DHCP server does not answer Pimox worker VM DHCP requests behind `vmbr0`: `LAB_PIMOX_WORKER_NETWORK_MODE=static`. DHCP mode remains available when LAN DHCP is known to answer VM traffic: set `LAB_PIMOX_WORKER_NETWORK_MODE=dhcp`. Static mode uses `LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, `LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, `LAB_PIMOX_WORKER_INTERFACE=enp0s18`, and `LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` only when the Pimox host firewall already permits bridged worker IP traffic. Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under `${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. When another process holds the xtables lock, Jeannie waits up to `${LAB_PIMOX_WORKER_IPTABLES_WAIT_SECONDS:-60}` seconds before failing. Worker NIC MAC addresses default to the VMID-derived deterministic format used by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` only when Pimox-generated MAC addresses are preferred. Worker clone failures are intentionally faster than template build failures: `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, and `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600` by default. The broader `LAB_PIMOX_GUEST_TIMEOUT_SECONDS` still controls template preparation. On the Orange Pi 5 Plus with Pimox 7 ARM, Jeannie launches the generated KVM command under host CPU set `4-7`. This keeps the guest's secondary CPU on the compatible RK3588 cores during PSCI boot. The same setting is used for the template build and worker clones, and each worker must report its configured CPU count through SSH before it is added to the Kubernetes worker inventory. Set `LAB_PIMOX_KVM_CPUSET=` to use the normal `qm start` path on a different Pimox host, or override template and worker values separately with `LAB_PIMOX_TEMPLATE_KVM_CPUSET` and `LAB_PIMOX_WORKER_KVM_CPUSET`. Static guest-network validation skips the extra LAN ICMP probe by default because the ISP modem may not answer worker VM pings. Set `LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host should be required during qemu-guest-agent network setup. After qemu-guest-agent reports an IP, Debian skips the extra host-side ICMP probe by default and lets SSH be the reachability check. Enable the host-side ICMP gate with `LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true`. Worker indexes are stable: index `1` maps to VMID `9010`, `pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, `pimox-worker-02`, and worker key `pimox02`. `LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to empty, so the pipeline owns index `1` unless you set `LAB_PIMOX_SKIP_WORKER_INDEXES=1`. Useful overrides: ```bash ./jeannie rebuild-cluster LAB_PIMOX_PIPELINE=false ./jeannie up LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=true ./jeannie up LAB_PIMOX_WORKER_COUNT=0 ./jeannie up LAB_PIMOX_WORKER_COUNT=2 ./jeannie up LAB_PIMOX_SKIP_WORKER_INDEXES=1 LAB_PIMOX_WORKER_COUNT=2 ./jeannie up LAB_PIMOX_WORKER_BASE_VMID=9020 ./jeannie up LAB_PIMOX_WORKER_STORAGE= ./jeannie up LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./jeannie up LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./jeannie up ``` ## OpenWrt firewall VM OpenWrt is not built from the Debian golden-node template. The Kubernetes template remains Debian-only; OpenWrt uses the upstream ARM SystemReady `armsr/armv8` combined EFI image instead. The OpenWrt path is separate from the Kubernetes worker template. Run it only after `vmbr1` exists on the Pimox host and the second NIC/LAN side is safe to use: ```bash ./jeannie openwrt ``` Defaults: - VMID `9100` - VM name `openwrt-firewall` - disk storage `opi5_ssd` when that Pimox storage is available - WAN bridge `vmbr0` - LAN bridge `vmbr1` - LAN address `192.168.50.1/24` - LAN DHCP disabled by default - VM start disabled by default - VM onboot disabled by default - OpenWrt version `24.10.6` Useful overrides: ```bash LAB_OPENWRT_VMID=9100 LAB_OPENWRT_STORAGE= LAB_OPENWRT_WAN_BRIDGE=vmbr0 LAB_OPENWRT_LAN_BRIDGE=vmbr1 LAB_OPENWRT_LAN_IP=192.168.50.1 LAB_OPENWRT_LAN_NETMASK=255.255.255.0 LAB_OPENWRT_LAN_DHCP_ENABLED=false LAB_OPENWRT_START=false LAB_OPENWRT_ONBOOT=false LAB_OPENWRT_VERSION=24.10.6 LAB_OPENWRT_IMAGE_URL=https://downloads.openwrt.org/releases/24.10.6/targets/armsr/armv8/openwrt-24.10.6-armsr-armv8-generic-ext4-combined-efi.img.gz ``` The pipeline validates `vmbr0`, `vmbr1`, and the selected storage on the Pimox host. It refuses `local` as OpenWrt storage and refuses to create or modify host network bridges.