my-homelab-configs/bootstrap/provisioning

Homelab Provisioning Layer

This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus.

./jeannie up drives this layer in auto mode when the Pimox host is reachable at 192.168.100.80, qm is installed, and the existing bridge vmbr0 is present. The automation creates VM definitions only; it validates the bridge and refuses to edit Orange Pi host networking.

What It Installs

• dnsmasq for proxyDHCP and TFTP
• nginx for preseed, installer, and guest-prep scripts
• Debian 13 arm64 netboot assets under /opt/homelab-provisioning
• a preseed file for unattended Debian install
• a GRUB PXE rescue menu and HTTP rescue page at /rescue/
• guest prep scripts that install Kubernetes tools, containerd, qemu guest agent, cloud-init, OpenEBS dependencies, cgroup prerequisites, swap disablement, and the local registry trust path
• Kubernetes tools from the pinned minor in TF_VAR_kubernetes_minor_version, defaulting to v1.36
• containerd.io from Docker's Debian repository so worker runtimes use the same package source as the Debian control-plane host
• kernel boot options for cgroup support through TF_VAR_kernel_cgroup_boot_options
• a template sealing script at /usr/local/sbin/homelab-prepare-template.sh inside the installed VM that verifies cgroup boot state before sealing

Apply From Debian

The normal path is:

cd ~/my-homelab-configs
./jeannie up

For manual provisioning-only testing, find the LAN interface on the Debian server:

ip -br addr

Apply the provisioning layer:

cd ~/my-homelab-configs
tofu -chdir=bootstrap/provisioning init
TF_VAR_provisioning_interface=enp1s0 tofu -chdir=bootstrap/provisioning apply

Override TF_VAR_provisioning_interface with the interface that serves the Pimox VM network.

The default VM user is jv. The account uses /home/jv/.ssh/id_ed25519.pub from the Debian server when that key exists, and the password is locked by default. Set TF_VAR_template_user_ssh_authorized_keys or TF_VAR_template_user_password_hash before applying if you want different access.

Clones should get their intended hostname through cloud-init or the later clone automation. If they boot with the template hostname, the first-boot service generates a unique fallback name using TF_VAR_clone_hostname_prefix.

Pimox VM Template Flow

Create an arm64 VM in Pimox with UEFI firmware, a virtio disk, and a NIC on the same LAN as the Debian provisioning host. Put network boot first.

PXE should load grubaa64.efi and display a GRUB menu. The default entry keeps the automated template flow intact: it boots the Debian installer, fetches the preseed from http://192.168.100.73:8088/preseed/debian13-arm64-worker.cfg, and installs the golden image.

The same menu also includes rescue-oriented entries:

• Debian 13 arm64 rescue shell
• Debian 13 arm64 expert/manual install
• Debian 13 arm64 automated install, verbose
• UEFI firmware settings
• Exit PXE and continue local boot
• Reboot

The Debian provisioning host also publishes a rescue landing page:

curl http://192.168.100.73:8088/rescue/

If your Pimox firmware needs a different Debian arm64 EFI loader, override TF_VAR_pxe_boot_file.

After the first successful boot, run this inside the VM before converting it to a template:

sudo /usr/local/sbin/homelab-prepare-template.sh
sudo poweroff

Convert the powered-off VM to a Pimox template. Clone it for each new worker, set a unique hostname and IP address through cloud-init or DHCP reservation, then add it to bootstrap/cluster/variables.tf or a .tfvars file:

worker_nodes = {
  pimox01 = {
    host         = "192.168.100.90"
    user         = "jv"
    node_name    = "pimox01"
    ssh_key_path = "/home/jv/.ssh/id_ed25519"
  }
}

Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH.

Pimox Automation

./jeannie up runs Pimox provisioning by default and sets TF_VAR_pimox_template_builder_enabled=true when Pimox is reachable. The layer SSHes into the Pimox host, create the template-build VM with qm, boot it from PXE, wait for the installed VM over SSH, run /usr/local/sbin/homelab-prepare-template.sh, power it off, switch boot order back to disk first, and run qm template.

The template sealing step discovers the installed VM IP through qemu-guest-agent, so a DHCP reservation is no longer required for the temporary template-build VM. If you still want to force a known address, set TF_VAR_pimox_template_build_host. Leave it unset for the normal autodiscovery path.

./jeannie up

Defaults match the observed Pimox template VM shape: OVMF firmware, virtio networking, virtio-scsi disk, vmbr0, local template storage, 1 socket with 2 cores, 8 GiB memory, and explicit args: -smp 2,sockets=1,cores=2,threads=1. Override TF_VAR_pimox_template_scsi0, TF_VAR_pimox_template_efidisk0, TF_VAR_pimox_template_cores, or TF_VAR_pimox_template_memory if the Orange Pi template layout changes.

./jeannie up also creates or reuses worker clones after the template exists. It defaults to one worker, VMID 9010, name pimox-worker-01, deterministic locally administered MAC address, 1 socket with 2 cores, 8 GiB RAM, opi5_ssd clone storage, and qemu-guest-agent IP discovery when that Pimox storage is available. New workers are full clones created with qm clone --storage, so the template can remain on local while worker disks land on the Orange Pi SSD-backed non-local Pimox datastore. Keep durable data on the HP Debian laptop's data-vg or external backups and treat Orange Pi VM disks as rebuildable worker capacity. Set LAB_PIMOX_WORKER_REPLACE_EXISTING=true to destroy and recreate existing worker VMs from the current template. The pipeline refuses LAB_PIMOX_WORKER_STORAGE=local so only the template VM lives on local storage.

Worker networking defaults to static because the ISP modem DHCP server does not answer Pimox worker VM DHCP requests behind vmbr0: LAB_PIMOX_WORKER_NETWORK_MODE=static. DHCP mode remains available when LAN DHCP is known to answer VM traffic: set LAB_PIMOX_WORKER_NETWORK_MODE=dhcp. Static mode uses LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76", LAB_PIMOX_WORKER_GATEWAY=192.168.100.1, LAB_PIMOX_WORKER_INTERFACE=enp0s18, and LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1". For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions for each worker IP in DOCKER-USER when that chain exists, or FORWARD otherwise. If iptables is missing on the Pimox host, Jeannie installs it before adding those rules. Disable this with LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false only when the Pimox host firewall already permits bridged worker IP traffic. Before adding rules, Jeannie saves iptables-save output on the Pimox host under ${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}. When another process holds the xtables lock, Jeannie waits up to ${LAB_PIMOX_WORKER_IPTABLES_WAIT_SECONDS:-60} seconds before failing. Worker NIC MAC addresses default to the VMID-derived deterministic format used by the original working Pimox worker path. Set LAB_PIMOX_WORKER_MAC_MODE=auto only when Pimox-generated MAC addresses are preferred. Worker clone failures are intentionally faster than template build failures: LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300, LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600, and LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600 by default. The broader LAB_PIMOX_GUEST_TIMEOUT_SECONDS still controls template preparation. Static guest-network validation skips the extra LAN ICMP probe by default because the ISP modem may not answer worker VM pings. Set LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true only when the gateway/probe host should be required during qemu-guest-agent network setup. After qemu-guest-agent reports an IP, Debian skips the extra host-side ICMP probe by default and lets SSH be the reachability check. Enable the host-side ICMP gate with LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true.

Worker indexes are stable: index 1 maps to VMID 9010, pimox-worker-01, and worker key pimox01; index 2 maps to VMID 9011, pimox-worker-02, and worker key pimox02. LAB_PIMOX_SKIP_WORKER_INDEXES defaults to empty, so the pipeline owns index 1 unless you set LAB_PIMOX_SKIP_WORKER_INDEXES=1.

Useful overrides:

./jeannie rebuild-cluster
LAB_PIMOX_PIPELINE=false ./jeannie up
LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_WORKER_COUNT=0 ./jeannie up
LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_SKIP_WORKER_INDEXES=1 LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_WORKER_BASE_VMID=9020 ./jeannie up
LAB_PIMOX_WORKER_STORAGE=<stable-pimox-storage> ./jeannie up
LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./jeannie up

OpenWrt firewall VM

OpenWrt is not built from the Debian golden-node template. The Kubernetes template remains Debian-only; OpenWrt uses the upstream ARM SystemReady armsr/armv8 combined EFI image instead.

The OpenWrt path is separate from the Kubernetes worker template. Run it only after vmbr1 exists on the Pimox host and the second NIC/LAN side is safe to use:

./jeannie openwrt

Defaults:

• VMID 9100
• VM name openwrt-firewall
• disk storage opi5_ssd when that Pimox storage is available
• WAN bridge vmbr0
• LAN bridge vmbr1
• LAN address 192.168.50.1/24
• LAN DHCP disabled by default
• VM start disabled by default
• VM onboot disabled by default
• OpenWrt version 24.10.6

Useful overrides:

LAB_OPENWRT_VMID=9100
LAB_OPENWRT_STORAGE=<stable-pimox-storage>
LAB_OPENWRT_WAN_BRIDGE=vmbr0
LAB_OPENWRT_LAN_BRIDGE=vmbr1
LAB_OPENWRT_LAN_IP=192.168.50.1
LAB_OPENWRT_LAN_NETMASK=255.255.255.0
LAB_OPENWRT_LAN_DHCP_ENABLED=false
LAB_OPENWRT_START=false
LAB_OPENWRT_ONBOOT=false
LAB_OPENWRT_VERSION=24.10.6
LAB_OPENWRT_IMAGE_URL=https://downloads.openwrt.org/releases/24.10.6/targets/armsr/armv8/openwrt-24.10.6-armsr-armv8-generic-ext4-combined-efi.img.gz

The pipeline validates vmbr0, vmbr1, and the selected storage on the Pimox host. It refuses local as OpenWrt storage and refuses to create or modify host network bridges.