8.4 KiB

Raw Blame History

Homelab Provisioning Layer

This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus.

./jeannie up drives this layer in auto mode when the Pimox host is reachable at 192.168.100.80, qm is installed, and the existing bridge vmbr0 is present. The automation creates VM definitions only; it validates the bridge and refuses to edit Orange Pi host networking.

What It Installs

dnsmasq for proxyDHCP and TFTP
nginx for preseed, installer, and guest-prep scripts
Debian 13 arm64 netboot assets under /opt/homelab-provisioning
a preseed file for unattended Debian install
a GRUB PXE rescue menu and HTTP rescue page at /rescue/
guest prep scripts that install Kubernetes tools, containerd, qemu guest agent, cloud-init, OpenEBS dependencies, cgroup prerequisites, swap disablement, and the local registry trust path
Kubernetes tools from the pinned minor in TF_VAR_kubernetes_minor_version, defaulting to v1.36
containerd.io from Docker's Debian repository so worker runtimes use the same package source as the Debian control-plane host
kernel boot options for cgroup support through TF_VAR_kernel_cgroup_boot_options
a template sealing script at /usr/local/sbin/homelab-prepare-template.sh inside the installed VM that verifies cgroup boot state before sealing

Apply From Debian

The normal path is:

cd ~/my-homelab-configs
./jeannie up

For manual provisioning-only testing, find the LAN interface on the Debian server:

ip -br addr

Apply the provisioning layer:

cd ~/my-homelab-configs
tofu -chdir=bootstrap/provisioning init
TF_VAR_provisioning_interface=enp1s0 tofu -chdir=bootstrap/provisioning apply

Override TF_VAR_provisioning_interface with the interface that serves the Pimox VM network.

The default VM user is jv. The account uses /home/jv/.ssh/id_ed25519.pub from the Debian server when that key exists, and the password is locked by default. Set TF_VAR_template_user_ssh_authorized_keys or TF_VAR_template_user_password_hash before applying if you want different access.

Clones should get their intended hostname through cloud-init or the later clone automation. If they boot with the template hostname, the first-boot service generates a unique fallback name using TF_VAR_clone_hostname_prefix.

Pimox VM Template Flow

Create an arm64 VM in Pimox with UEFI firmware, a virtio disk, and a NIC on the same LAN as the Debian provisioning host. Put network boot first.

PXE should load grubaa64.efi and display a GRUB menu. The default entry keeps the automated template flow intact: it boots the Debian installer, fetches the preseed from http://192.168.100.73:8088/preseed/debian13-arm64-worker.cfg, and installs the golden image.

The same menu also includes rescue-oriented entries:

Debian 13 arm64 rescue shell
Debian 13 arm64 expert/manual install
Debian 13 arm64 automated install, verbose
UEFI firmware settings
Exit PXE and continue local boot
Reboot

The Debian provisioning host also publishes a rescue landing page:

curl http://192.168.100.73:8088/rescue/

If your Pimox firmware needs a different Debian arm64 EFI loader, override TF_VAR_pxe_boot_file.

After the first successful boot, run this inside the VM before converting it to a template:

sudo /usr/local/sbin/homelab-prepare-template.sh
sudo poweroff

Convert the powered-off VM to a Pimox template. Clone it for each new worker, set a unique hostname and IP address through cloud-init or DHCP reservation, then add it to bootstrap/cluster/variables.tf or a .tfvars file:

worker_nodes = {
  pimox01 = {
    host         = "192.168.100.90"
    user         = "jv"
    node_name    = "pimox01"
    ssh_key_path = "/home/jv/.ssh/id_ed25519"
  }
}

Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH.

Pimox Automation

./jeannie up runs Pimox provisioning by default and sets TF_VAR_pimox_template_builder_enabled=true when Pimox is reachable. The layer SSHes into the Pimox host, create the template-build VM with qm, boot it from PXE, wait for the installed VM over SSH, run /usr/local/sbin/homelab-prepare-template.sh, power it off, switch boot order back to disk first, and run qm template.

The template sealing step discovers the installed VM IP through qemu-guest-agent, so a DHCP reservation is no longer required for the temporary template-build VM. If you still want to force a known address, set TF_VAR_pimox_template_build_host.

./jeannie up

Defaults match the observed Pimox template VM shape: OVMF firmware, virtio networking, virtio-scsi disk, vmbr0, local template storage, 1 socket with 2 cores, and 4 GiB memory. Override TF_VAR_pimox_template_scsi0, TF_VAR_pimox_template_efidisk0, TF_VAR_pimox_template_cores, or TF_VAR_pimox_template_memory if the Orange Pi template layout changes.

./jeannie up also creates or reuses worker clones after the template exists. It defaults to one worker, VMID 9010, name pimox-worker-01, deterministic locally administered MAC address, 1 socket with 2 cores, 4 GiB RAM, opi5_ssd clone storage, and qemu-guest-agent IP discovery when that Pimox storage is available. New workers are full clones created with qm clone --storage, so the template can remain on local while worker disks land on the Orange Pi SSD-backed non-local Pimox datastore. Keep durable data on the HP Debian laptop's data-vg or external backups and treat Orange Pi VM disks as rebuildable worker capacity. Set LAB_PIMOX_WORKER_REPLACE_EXISTING=true to destroy and recreate existing worker VMs from the current template. The pipeline refuses LAB_PIMOX_WORKER_STORAGE=local so only the template VM lives on local storage.

Worker networking defaults to deterministic static LAN addresses configured through qemu-guest-agent after clone/start and before SSH wait: LAB_PIMOX_WORKER_NETWORK_MODE=static, LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76", LAB_PIMOX_WORKER_GATEWAY=192.168.100.1, LAB_PIMOX_WORKER_INTERFACE=enp0s18, and LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1". Use LAB_PIMOX_WORKER_NETWORK_MODE=dhcp only when DHCP reservations are known to be stable.

Worker indexes are stable: index 1 maps to VMID 9010, pimox-worker-01, and worker key pimox01; index 2 maps to VMID 9011, pimox-worker-02, and worker key pimox02. LAB_PIMOX_SKIP_WORKER_INDEXES defaults to empty, so the pipeline owns index 1 unless you set LAB_PIMOX_SKIP_WORKER_INDEXES=1.

Useful overrides:

./jeannie rebuild-cluster
LAB_PIMOX_PIPELINE=false ./jeannie up
LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_WORKER_COUNT=0 ./jeannie up
LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_SKIP_WORKER_INDEXES=1 LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_WORKER_BASE_VMID=9020 ./jeannie up
LAB_PIMOX_WORKER_STORAGE=<stable-pimox-storage> ./jeannie up
LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./jeannie up

OpenWrt firewall VM

OpenWrt is not built from the Debian golden-node template. The Kubernetes template remains Debian-only; OpenWrt uses the upstream ARM SystemReady armsr/armv8 combined EFI image instead.

The OpenWrt path is separate from the Kubernetes worker template. Run it only after vmbr1 exists on the Pimox host and the second NIC/LAN side is safe to use:

./jeannie openwrt

Defaults:

VMID 9100
VM name openwrt-firewall
disk storage opi5_ssd when that Pimox storage is available
WAN bridge vmbr0
LAN bridge vmbr1
LAN address 192.168.50.1/24
LAN DHCP disabled by default
VM start disabled by default
VM onboot disabled by default
OpenWrt version 24.10.6