178 lines
6.6 KiB
Markdown
178 lines
6.6 KiB
Markdown
# Homelab Provisioning Layer
|
|
|
|
This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus.
|
|
|
|
`./lab.sh up` drives this layer in auto mode when the Pimox host is reachable at
|
|
`192.168.100.80`, `qm` is installed, and the existing bridge `vmbr0` is present.
|
|
The automation creates VM definitions only; it validates the bridge and refuses
|
|
to edit Orange Pi host networking.
|
|
|
|
## What It Installs
|
|
|
|
- `dnsmasq` for proxyDHCP and TFTP
|
|
- `nginx` for preseed, installer, and guest-prep scripts
|
|
- Debian 13 arm64 netboot assets under `/opt/homelab-provisioning`
|
|
- a preseed file for unattended Debian install
|
|
- guest prep scripts that install Kubernetes tools, containerd, qemu guest agent, cloud-init, OpenEBS dependencies, cgroup prerequisites, swap disablement, and the local registry trust path
|
|
- kernel boot options for cgroup support through `TF_VAR_kernel_cgroup_boot_options`
|
|
- a template sealing script at `/usr/local/sbin/homelab-prepare-template.sh` inside the installed VM that verifies cgroup boot state before sealing
|
|
|
|
## Apply From Debian
|
|
|
|
The normal path is:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
./lab.sh up
|
|
```
|
|
|
|
For manual provisioning-only testing, find the LAN interface on the Debian
|
|
server:
|
|
|
|
```bash
|
|
ip -br addr
|
|
```
|
|
|
|
Apply the provisioning layer:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
tofu -chdir=bootstrap/provisioning init
|
|
TF_VAR_provisioning_interface=enp1s0 tofu -chdir=bootstrap/provisioning apply
|
|
```
|
|
|
|
Override `TF_VAR_provisioning_interface` with the interface that serves the Pimox VM network.
|
|
|
|
The default VM user is `jv`. The account uses `/home/jv/.ssh/id_ed25519.pub`
|
|
from the Debian server when that key exists, and the password is locked by
|
|
default. Set `TF_VAR_template_user_ssh_authorized_keys` or
|
|
`TF_VAR_template_user_password_hash` before applying if you want different
|
|
access.
|
|
|
|
Clones should get their intended hostname through cloud-init or the later clone
|
|
automation. If they boot with the template hostname, the first-boot service
|
|
generates a unique fallback name using `TF_VAR_clone_hostname_prefix`.
|
|
|
|
## Pimox VM Template Flow
|
|
|
|
Create an arm64 VM in Pimox with UEFI firmware, a virtio disk, and a NIC on the same LAN as the Debian provisioning host. Put network boot first.
|
|
|
|
PXE should load `grubaa64.efi`, boot the Debian installer, fetch the preseed from `http://192.168.100.68:8088/preseed/debian13-arm64-worker.cfg`, and install the golden image.
|
|
|
|
If your Pimox firmware needs a different Debian arm64 EFI loader, override `TF_VAR_pxe_boot_file`.
|
|
|
|
After the first successful boot, run this inside the VM before converting it to a template:
|
|
|
|
```bash
|
|
sudo /usr/local/sbin/homelab-prepare-template.sh
|
|
sudo poweroff
|
|
```
|
|
|
|
Convert the powered-off VM to a Pimox template. Clone it for each new worker, set a unique hostname and IP address through cloud-init or DHCP reservation, then add it to `bootstrap/cluster/variables.tf` or a `.tfvars` file:
|
|
|
|
```hcl
|
|
worker_nodes = {
|
|
pimox01 = {
|
|
host = "192.168.100.90"
|
|
user = "jv"
|
|
node_name = "pimox01"
|
|
ssh_key_path = "/home/jv/.ssh/id_ed25519"
|
|
}
|
|
}
|
|
```
|
|
|
|
Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH.
|
|
|
|
## Pimox Automation
|
|
|
|
`./lab.sh up` runs Pimox provisioning by default and sets
|
|
`TF_VAR_pimox_template_builder_enabled=true` when Pimox is reachable. The layer SSHes into the
|
|
Pimox host, create the template-build VM with `qm`, boot it from PXE, wait for
|
|
the installed VM over SSH, run `/usr/local/sbin/homelab-prepare-template.sh`,
|
|
power it off, switch boot order back to disk first, and run `qm template`.
|
|
|
|
The template sealing step discovers the installed VM IP through
|
|
qemu-guest-agent, so a DHCP reservation is no longer required for the temporary
|
|
template-build VM. If you still want to force a known address, set
|
|
`TF_VAR_pimox_template_build_host`.
|
|
|
|
```bash
|
|
./lab.sh up
|
|
```
|
|
|
|
Defaults match the observed Pimox template VM shape: OVMF firmware, virtio
|
|
networking, virtio-scsi disk, `vmbr0`, `local` template storage, 1 socket with
|
|
2 cores, and 4 GiB memory. Override `TF_VAR_pimox_template_scsi0`,
|
|
`TF_VAR_pimox_template_efidisk0`, `TF_VAR_pimox_template_cores`, or
|
|
`TF_VAR_pimox_template_memory` if the Orange Pi template layout changes.
|
|
|
|
`./lab.sh up` also creates or reuses worker clones after the template exists. It
|
|
defaults to one worker, VMID `9010`, name `pimox-worker-01`, deterministic
|
|
locally administered MAC address, 1 socket
|
|
with 2 cores, 4 GiB RAM, `nvme_thin_pool` clone storage, and qemu-guest-agent IP
|
|
discovery. New workers are full clones created with
|
|
`qm clone --storage`, so the template can remain on `local` while worker disks
|
|
land on the NVMe thin pool. Set `LAB_PIMOX_WORKER_REPLACE_EXISTING=true` to
|
|
destroy and recreate existing worker VMs from the current template. The pipeline
|
|
refuses `LAB_PIMOX_WORKER_STORAGE=local` so only the template VM lives on local
|
|
storage. Useful overrides:
|
|
|
|
```bash
|
|
./lab.sh rebuild-cluster
|
|
LAB_PIMOX_PIPELINE=false ./lab.sh up
|
|
LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=true ./lab.sh up
|
|
LAB_PIMOX_WORKER_COUNT=0 ./lab.sh up
|
|
LAB_PIMOX_WORKER_COUNT=2 ./lab.sh up
|
|
LAB_PIMOX_WORKER_BASE_VMID=9020 ./lab.sh up
|
|
LAB_PIMOX_WORKER_STORAGE=nvme_thin_pool ./lab.sh up
|
|
LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./lab.sh up
|
|
LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./lab.sh up
|
|
```
|
|
|
|
## OpenWrt firewall VM
|
|
|
|
OpenWrt is not built from the Debian golden-node template. The Kubernetes
|
|
template remains Debian-only; OpenWrt uses the upstream ARM SystemReady
|
|
`armsr/armv8` combined EFI image instead.
|
|
|
|
The OpenWrt path is separate from the Kubernetes worker template. Run it only
|
|
after `vmbr1` exists on the Pimox host and the second NIC/LAN side is safe to
|
|
use:
|
|
|
|
```bash
|
|
./lab.sh openwrt
|
|
```
|
|
|
|
Defaults:
|
|
|
|
- VMID `9100`
|
|
- VM name `openwrt-firewall`
|
|
- disk storage `nvme_thin_pool`
|
|
- WAN bridge `vmbr0`
|
|
- LAN bridge `vmbr1`
|
|
- LAN address `192.168.50.1/24`
|
|
- LAN DHCP disabled by default
|
|
- VM start disabled by default
|
|
- VM onboot disabled by default
|
|
- OpenWrt version `24.10.6`
|
|
|
|
Useful overrides:
|
|
|
|
```bash
|
|
LAB_OPENWRT_VMID=9100
|
|
LAB_OPENWRT_STORAGE=nvme_thin_pool
|
|
LAB_OPENWRT_WAN_BRIDGE=vmbr0
|
|
LAB_OPENWRT_LAN_BRIDGE=vmbr1
|
|
LAB_OPENWRT_LAN_IP=192.168.50.1
|
|
LAB_OPENWRT_LAN_NETMASK=255.255.255.0
|
|
LAB_OPENWRT_LAN_DHCP_ENABLED=false
|
|
LAB_OPENWRT_START=false
|
|
LAB_OPENWRT_ONBOOT=false
|
|
LAB_OPENWRT_VERSION=24.10.6
|
|
LAB_OPENWRT_IMAGE_URL=https://downloads.openwrt.org/releases/24.10.6/targets/armsr/armv8/openwrt-24.10.6-armsr-armv8-generic-ext4-combined-efi.img.gz
|
|
```
|
|
|
|
The pipeline validates `vmbr0`, `vmbr1`, and `nvme_thin_pool` on the Pimox host.
|
|
It refuses `local` as OpenWrt storage and refuses to create or modify host
|
|
network bridges.
|