jv
/
my-homelab-configs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
my-homelab-configs/bootstrap/provisioning/README.md

246 lines
10 KiB
Markdown
Raw Blame History

# Homelab Provisioning Layer
This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus.
`./jeannie up` drives this layer in auto mode when the Pimox host is reachable at
`192.168.100.80`, `qm` is installed, and the existing bridge `vmbr0` is present.
The automation creates VM definitions only; it validates the bridge and refuses
to edit Orange Pi host networking.
## What It Installs
- `dnsmasq` for proxyDHCP and TFTP
- `nginx` for preseed, installer, and guest-prep scripts
- Debian 13 arm64 netboot assets under `/opt/homelab-provisioning`
- a preseed file for unattended Debian install
- a GRUB PXE rescue menu and HTTP rescue page at `/rescue/`
- guest prep scripts that install Kubernetes tools, containerd, qemu guest agent, cloud-init, OpenEBS dependencies, cgroup prerequisites, swap disablement, and the local registry trust path
- Kubernetes tools from the pinned minor in `TF_VAR_kubernetes_minor_version`, defaulting to `v1.36`
- `containerd.io` from Docker's Debian repository so worker runtimes use the
same package source as the Debian control-plane host
- kernel boot options for cgroup support through `TF_VAR_kernel_cgroup_boot_options`
- a template sealing script at `/usr/local/sbin/homelab-prepare-template.sh` inside the installed VM that verifies cgroup boot state before sealing
## Apply From Debian
The normal path is:
```bash
cd ~/my-homelab-configs
./jeannie up
```
For manual provisioning-only testing, find the LAN interface on the Debian
server:
```bash
ip -br addr
```
Apply the provisioning layer:
```bash
cd ~/my-homelab-configs
tofu -chdir=bootstrap/provisioning init
TF_VAR_provisioning_interface=enp1s0 tofu -chdir=bootstrap/provisioning apply
```
Override `TF_VAR_provisioning_interface` with the interface that serves the Pimox VM network.
The default VM user is `jv`. The account uses `/home/jv/.ssh/id_ed25519.pub`
from the Debian server when that key exists, and the password is locked by
default. Set `TF_VAR_template_user_ssh_authorized_keys` or
`TF_VAR_template_user_password_hash` before applying if you want different
access.
Clones should get their intended hostname through cloud-init or the later clone
automation. If they boot with the template hostname, the first-boot service
generates a unique fallback name using `TF_VAR_clone_hostname_prefix`.
## Pimox VM Template Flow
Create an arm64 VM in Pimox with UEFI firmware, a virtio disk, and a NIC on the same LAN as the Debian provisioning host. Put network boot first.
PXE should load `grubaa64.efi` and display a GRUB menu. The default entry keeps
the automated template flow intact: it boots the Debian installer, fetches the
preseed from `http://192.168.100.73:8088/preseed/debian13-arm64-worker.cfg`, and
installs the golden image.
The same menu also includes rescue-oriented entries:
- Debian 13 arm64 rescue shell
- Debian 13 arm64 expert/manual install
- Debian 13 arm64 automated install, verbose
- UEFI firmware settings
- Exit PXE and continue local boot
- Reboot
The Debian provisioning host also publishes a rescue landing page:
```bash
curl http://192.168.100.73:8088/rescue/
```
If your Pimox firmware needs a different Debian arm64 EFI loader, override `TF_VAR_pxe_boot_file`.
After the first successful boot, run this inside the VM before converting it to a template:
```bash
sudo /usr/local/sbin/homelab-prepare-template.sh
sudo poweroff
```
Convert the powered-off VM to a Pimox template. Clone it for each new worker, set a unique hostname and IP address through cloud-init or DHCP reservation, then add it to `bootstrap/cluster/variables.tf` or a `.tfvars` file:
```hcl
worker_nodes = {
pimox01 = {
host = "192.168.100.90"
user = "jv"
node_name = "pimox01"
ssh_key_path = "/home/jv/.ssh/id_ed25519"
}
}
```
Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH.
## Pimox Automation
`./jeannie up` runs Pimox provisioning by default and sets
`TF_VAR_pimox_template_builder_enabled=true` when Pimox is reachable. The layer SSHes into the
Pimox host, create the template-build VM with `qm`, boot it from PXE, wait for
the installed VM over SSH, run `/usr/local/sbin/homelab-prepare-template.sh`,
power it off, switch boot order back to disk first, and run `qm template`.
The template sealing step discovers the installed VM IP through
qemu-guest-agent, so a DHCP reservation is no longer required for the temporary
template-build VM. If you still want to force a known address, set
`TF_VAR_pimox_template_build_host`.
```bash
./jeannie up
```
Defaults match the observed Pimox template VM shape: OVMF firmware, virtio
networking, virtio-scsi disk, `vmbr0`, `local` template storage, 1 socket with
2 cores, 8 GiB memory, and explicit `args: -smp 2,sockets=1,cores=2,threads=1`.
Override `TF_VAR_pimox_template_scsi0`,
`TF_VAR_pimox_template_efidisk0`, `TF_VAR_pimox_template_cores`, or
`TF_VAR_pimox_template_memory` if the Orange Pi template layout changes.
`./jeannie up` also creates or reuses worker clones after the template exists. It
defaults to one worker, VMID `9010`, name `pimox-worker-01`, deterministic
locally administered MAC address, 1 socket
with 2 cores, 8 GiB RAM, `opi5_ssd` clone storage, and qemu-guest-agent IP
discovery when that Pimox storage is available. New workers are full clones
created with `qm clone --storage`, so the template can remain on `local` while
worker disks land on the Orange Pi SSD-backed non-local Pimox datastore. Keep
durable data on the HP Debian laptop's `data-vg` or external backups and treat
Orange Pi VM disks as rebuildable worker capacity. Set
`LAB_PIMOX_WORKER_REPLACE_EXISTING=true` to
destroy and recreate existing worker VMs from the current template. The pipeline
refuses `LAB_PIMOX_WORKER_STORAGE=local` so only the template VM lives on local
storage.
Worker networking defaults to static because the ISP modem DHCP server does not
answer Pimox worker VM DHCP requests behind `vmbr0`:
`LAB_PIMOX_WORKER_NETWORK_MODE=static`. DHCP mode remains available when LAN
DHCP is known to answer VM traffic: set `LAB_PIMOX_WORKER_NETWORK_MODE=dhcp`.
Static mode uses
`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`,
`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, `LAB_PIMOX_WORKER_INTERFACE=enp0s18`,
and `LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`.
For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions
for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD`
otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before
adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false`
only when the Pimox host firewall already permits bridged worker IP traffic.
Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under
`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`.
When another process holds the xtables lock, Jeannie waits up to
`${LAB_PIMOX_WORKER_IPTABLES_WAIT_SECONDS:-60}` seconds before failing.
Worker NIC MAC addresses default to the VMID-derived deterministic format used
by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto`
only when Pimox-generated MAC addresses are preferred.
Worker clone failures are intentionally faster than template build failures:
`LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`,
`LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, and
`LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600` by default. The broader
`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` still controls template preparation.
Static guest-network validation skips the extra LAN ICMP probe by default
because the ISP modem may not answer worker VM pings. Set
`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host
should be required during qemu-guest-agent network setup. After
qemu-guest-agent reports an IP, Debian skips the extra host-side ICMP probe by
default and lets SSH be the reachability check. Enable the host-side ICMP gate
with `LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true`.
Worker indexes are stable: index `1` maps to VMID `9010`,
`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`,
`pimox-worker-02`, and worker key `pimox02`. `LAB_PIMOX_SKIP_WORKER_INDEXES`
defaults to empty, so the pipeline owns index `1` unless you set
`LAB_PIMOX_SKIP_WORKER_INDEXES=1`.
Useful overrides:
```bash
./jeannie rebuild-cluster
LAB_PIMOX_PIPELINE=false ./jeannie up
LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_WORKER_COUNT=0 ./jeannie up
LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_SKIP_WORKER_INDEXES=1 LAB_PIMOX_WORKER_COUNT=2 ./jeannie up
LAB_PIMOX_WORKER_BASE_VMID=9020 ./jeannie up
LAB_PIMOX_WORKER_STORAGE=<stable-pimox-storage> ./jeannie up
LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./jeannie up
LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./jeannie up
```
## OpenWrt firewall VM
OpenWrt is not built from the Debian golden-node template. The Kubernetes
template remains Debian-only; OpenWrt uses the upstream ARM SystemReady
`armsr/armv8` combined EFI image instead.
The OpenWrt path is separate from the Kubernetes worker template. Run it only
after `vmbr1` exists on the Pimox host and the second NIC/LAN side is safe to
use:
```bash
./jeannie openwrt
```
Defaults:
- VMID `9100`
- VM name `openwrt-firewall`
- disk storage `opi5_ssd` when that Pimox storage is available
- WAN bridge `vmbr0`
- LAN bridge `vmbr1`
- LAN address `192.168.50.1/24`
- LAN DHCP disabled by default
- VM start disabled by default
- VM onboot disabled by default
- OpenWrt version `24.10.6`
Useful overrides:
```bash
LAB_OPENWRT_VMID=9100
LAB_OPENWRT_STORAGE=<stable-pimox-storage>
LAB_OPENWRT_WAN_BRIDGE=vmbr0
LAB_OPENWRT_LAN_BRIDGE=vmbr1
LAB_OPENWRT_LAN_IP=192.168.50.1
LAB_OPENWRT_LAN_NETMASK=255.255.255.0
LAB_OPENWRT_LAN_DHCP_ENABLED=false
LAB_OPENWRT_START=false
LAB_OPENWRT_ONBOOT=false
LAB_OPENWRT_VERSION=24.10.6
LAB_OPENWRT_IMAGE_URL=https://downloads.openwrt.org/releases/24.10.6/targets/armsr/armv8/openwrt-24.10.6-armsr-armv8-generic-ext4-combined-efi.img.gz
```
The pipeline validates `vmbr0`, `vmbr1`, and the selected storage on the Pimox host.
It refuses `local` as OpenWrt storage and refuses to create or modify host
network bridges.
Reference in New Issue View Git Blame Copy Permalink