jv
/
my-homelab-configs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
my-homelab-configs/bootstrap/provisioning/README.md

261 lines
11 KiB
Markdown
Raw Blame History

# Homelab Provisioning Layer
This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus.
`./jeannie up` drives this layer in auto mode when the Pimox host is reachable at
`192.168.100.80`, `qm` is installed, and the existing bridge `vmbr0` is present.
The automation creates VM definitions only; it validates the bridge and refuses
to edit Orange Pi host networking.
## What It Installs
- `dnsmasq` for proxyDHCP and TFTP
- `nginx` for preseed, installer, and guest-prep scripts
- Debian 13 arm64 netboot assets under `/opt/homelab-provisioning`
- a preseed file for unattended Debian install
- a GRUB PXE rescue menu and HTTP rescue page at `/rescue/`
- guest prep scripts that install Kubernetes tools, containerd, qemu guest agent, cloud-init, OpenEBS dependencies, cgroup prerequisites, swap disablement, and the local registry trust path
- Kubernetes tools from the pinned minor in `TF_VAR_kubernetes_minor_version`, defaulting to `v1.36`
- `containerd.io` from Docker's Debian repository so worker runtimes use the
same package source as the Debian control-plane host
- kernel boot options for cgroup support through `TF_VAR_kernel_cgroup_boot_options`
- a template sealing script at `/usr/local/sbin/homelab-prepare-template.sh` inside the installed VM that verifies cgroup boot state before sealing
## Apply From Debian
The normal path is:
```bash
cd ~/my-homelab-configs
./jeannie up
```
For manual provisioning-only testing, find the LAN interface on the Debian
server:
```bash
ip -br addr
```
Apply the provisioning layer:
```bash
cd ~/my-homelab-configs
tofu -chdir=bootstrap/provisioning init
TF_VAR_provisioning_interface=enp1s0 tofu -chdir=bootstrap/provisioning apply
```
Override `TF_VAR_provisioning_interface` with the interface that serves the Pimox VM network.
The default VM user is `jv`. The account uses `/home/jv/.ssh/id_ed25519.pub`
from the Debian server when that key exists, and the password is locked by
default. Set `TF_VAR_template_user_ssh_authorized_keys` or
`TF_VAR_template_user_password_hash` before applying if you want different
access.
Clones should get their intended hostname through cloud-init or the later clone
automation. If they boot with the template hostname, the first-boot service
generates a unique fallback name using `TF_VAR_clone_hostname_prefix`.
## Pimox VM Template Flow
Create an arm64 VM in Pimox with UEFI firmware, a virtio disk, and a NIC on the same LAN as the Debian provisioning host. Put network boot first.
PXE should load `grubaa64.efi` and display a GRUB menu. The default entry keeps
the automated template flow intact: it boots the Debian installer, fetches the
preseed from `http://192.168.100.73:8088/preseed/debian13-arm64-worker.cfg`, and
installs the golden image.
The same menu also includes rescue-oriented entries:
- Debian 13 arm64 rescue shell
- Debian 13 arm64 expert/manual install
- Debian 13 arm64 automated install, verbose
- UEFI firmware settings
- Exit PXE and continue local boot
- Reboot
The Debian provisioning host also publishes a rescue landing page:
```bash
curl http://192.168.100.73:8088/rescue/
```
If your Pimox firmware needs a different Debian arm64 EFI loader, override `TF_VAR_pxe_boot_file`.
After the first successful boot, run this inside the VM before converting it to a template:
```bash
sudo /usr/local/sbin/homelab-prepare-template.sh
sudo poweroff
```
Convert the powered-off VM to a Pimox template. Clone it for each new worker, set a unique hostname and IP address through cloud-init or DHCP reservation, then add it to `bootstrap/cluster/variables.tf` or a `.tfvars` file:
```hcl
worker_nodes = {
pimox01 = {
host = "192.168.100.90"
user = "jv"
node_name = "pimox01"
ssh_key_path = "/home/jv/.ssh/id_ed25519"
}
}
```
Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH.
## Pimox Automation
`./jeannie up` runs Pimox provisioning by default and sets
`TF_VAR_pimox_template_builder_enabled=true` when Pimox is reachable. The layer SSHes into the
Pimox host, create the template-build VM with `qm`, boot it from PXE, wait for
the installed VM over SSH, run `/usr/local/sbin/homelab-prepare-template.sh`,
power it off, switch boot order back to disk first, and run `qm template`.
The template sealing step discovers the installed VM IP through
qemu-guest-agent, so a DHCP reservation is no longer required for the temporary
template-build VM. If you still want to force a known address, set
`TF_VAR_pimox_template_build_host`. Leave it unset for the normal autodiscovery
path. Once a guest IP or explicit guest host is known, the seal step now uses a
shorter SSH deadline of `TF_VAR_pimox_template_guest_ssh_timeout_seconds=300`
by default so bad IP/SSH assumptions fail quickly instead of consuming the full
template build timeout.
```bash
./jeannie up
```
Defaults match the observed Pimox template VM shape: OVMF firmware, virtio
networking, virtio-scsi disk, `vmbr0`, `local` template storage, 1 socket with
2 cores, and 8 GiB memory. The native `qm` CPU topology is the only `-smp`
argument; the provisioning stack removes legacy `args` values to prevent a
second conflicting `-smp` option.
Override `TF_VAR_pimox_template_scsi0`,
`TF_VAR_pimox_template_efidisk0`, `TF_VAR_pimox_template_cores`, or
`TF_VAR_pimox_template_memory` if the Orange Pi template layout changes.
`./jeannie up` also creates or reuses worker clones after the template exists. It
defaults to one worker, VMID `9010`, name `pimox-worker-01`, deterministic
locally administered MAC address, 1 socket
with 2 cores, 8 GiB RAM, `opi5_ssd` clone storage, and qemu-guest-agent IP
discovery when that Pimox storage is available. New workers are full clones
created with `qm clone --storage`, so the template can remain on `local` while
worker disks land on the Orange Pi SSD-backed non-local Pimox datastore. Keep
durable data on the HP Debian laptop's `data-vg` or external backups and treat
Orange Pi VM disks as rebuildable worker capacity. Set
`LAB_PIMOX_WORKER_REPLACE_EXISTING=true` to
destroy and recreate existing worker VMs from the current template. The pipeline
refuses `LAB_PIMOX_WORKER_STORAGE=local` so only the template VM lives on local
storage.
Worker networking defaults to static because the ISP modem DHCP server does not
answer Pimox worker VM DHCP requests behind `vmbr0`:
`LAB_PIMOX_WORKER_NETWORK_MODE=static`. DHCP mode remains available when LAN
DHCP is known to answer VM traffic: set `LAB_PIMOX_WORKER_NETWORK_MODE=dhcp`.
Static mode uses
`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`,
`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, `LAB_PIMOX_WORKER_INTERFACE=enp0s18`,
and `LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`.
For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions
for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD`
otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before
adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false`
only when the Pimox host firewall already permits bridged worker IP traffic.
Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under
`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`.
When another process holds the xtables lock, Jeannie waits up to
`${LAB_PIMOX_WORKER_IPTABLES_WAIT_SECONDS:-60}` seconds before failing.
Worker NIC MAC addresses default to the VMID-derived deterministic format used
by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto`
only when Pimox-generated MAC addresses are preferred.
Worker clone failures are intentionally faster than template build failures:
`LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`,
`LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, and
`LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600` by default. The broader
`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` still controls template preparation.
On the Orange Pi 5 Plus with Pimox 7 ARM, Jeannie launches the generated KVM
command under host CPU set `4-7`. This keeps the guest's secondary CPU on the
compatible RK3588 cores during PSCI boot. The same setting is used for the
template build and worker clones, and each worker must report its configured
CPU count through SSH before it is added to the Kubernetes worker inventory.
Set `LAB_PIMOX_KVM_CPUSET=` to use the normal `qm start` path on a different
Pimox host, or override template and worker values separately with
`LAB_PIMOX_TEMPLATE_KVM_CPUSET` and `LAB_PIMOX_WORKER_KVM_CPUSET`.
Static guest-network validation skips the extra LAN ICMP probe by default
because the ISP modem may not answer worker VM pings. Set
`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host
should be required during qemu-guest-agent network setup. After
qemu-guest-agent reports an IP, Debian skips the extra host-side ICMP probe by
default and lets SSH be the reachability check. Enable the host-side ICMP gate
with `LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true`.
Worker indexes are stable: index `1` maps to VMID `9010`,
`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`,
`pimox-worker-02`, and worker key `pimox02`. `LAB_PIMOX_SKIP_WORKER_INDEXES`
defaults to empty, so the pipeline owns index `1` unless you set
`LAB_PIMOX_SKIP_WORKER_INDEXES=1`.
Useful overrides:
```bash
./jeannie rebuild-cluster
LAB_PIMOX_PIPELINE=false ./jeannie up
LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_WORKER_COUNT=0 ./jeannie up
LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_SKIP_WORKER_INDEXES=1 LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_WORKER_BASE_VMID=9020 ./jeannie up
LAB_PIMOX_WORKER_STORAGE=<stable-pimox-storage> ./jeannie up
LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./jeannie up
```
## OpenWrt firewall VM
OpenWrt is not built from the Debian golden-node template. The Kubernetes
template remains Debian-only; OpenWrt uses the upstream ARM SystemReady
`armsr/armv8` combined EFI image instead.
The OpenWrt path is separate from the Kubernetes worker template. Run it only
after `vmbr1` exists on the Pimox host and the second NIC/LAN side is safe to
use:
```bash
./jeannie openwrt
```
Defaults:
- VMID `9100`
- VM name `openwrt-firewall`
- disk storage `opi5_ssd` when that Pimox storage is available
- WAN bridge `vmbr0`
- LAN bridge `vmbr1`
- LAN address `192.168.50.1/24`
- LAN DHCP disabled by default
- VM start disabled by default
- VM onboot disabled by default
- OpenWrt version `24.10.6`
Useful overrides:
```bash
LAB_OPENWRT_VMID=9100
LAB_OPENWRT_STORAGE=<stable-pimox-storage>
LAB_OPENWRT_WAN_BRIDGE=vmbr0
LAB_OPENWRT_LAN_BRIDGE=vmbr1
LAB_OPENWRT_LAN_IP=192.168.50.1
LAB_OPENWRT_LAN_NETMASK=255.255.255.0
LAB_OPENWRT_LAN_DHCP_ENABLED=false
LAB_OPENWRT_START=false
LAB_OPENWRT_ONBOOT=false
LAB_OPENWRT_VERSION=24.10.6
LAB_OPENWRT_IMAGE_URL=https://downloads.openwrt.org/releases/24.10.6/targets/armsr/armv8/openwrt-24.10.6-armsr-armv8-generic-ext4-combined-efi.img.gz
```
The pipeline validates `vmbr0`, `vmbr1`, and the selected storage on the Pimox host.
It refuses `local` as OpenWrt storage and refuses to create or modify host
network bridges.
Reference in New Issue View Git Blame Copy Permalink