Automate Pimox worker provisioning pipeline
This commit is contained in:
juvdiaz
parent
11ea473c7f
commit
df95e2ea5f
31
README.md
31
README.md
|
|
@ -29,11 +29,12 @@ accidentally modify the cluster.
|
|||
- serves Debian 13 arm64 netboot assets through TFTP and HTTP
|
||||
- creates a golden image install path with Kubernetes, containerd,
|
||||
qemu-guest-agent, cloud-init, and storage client packages ready
|
||||
- stays out of `./lab.sh up` so VM template creation remains manual
|
||||
- is driven by `./lab.sh up` when Pimox is reachable, without changing
|
||||
Orange Pi host networking
|
||||
|
||||
2. `bootstrap/cluster`
|
||||
- creates the kubeadm control plane on the Debian amd64 node
|
||||
- joins worker nodes such as Raspberry Pi arm64 nodes
|
||||
- joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes
|
||||
- configures Calico-compatible pod CIDR
|
||||
- configures containerd to pull from the in-cluster NodePort registry
|
||||
- creates retained host directories under `/var/openebs/local`
|
||||
|
|
@ -80,10 +81,19 @@ cd ~/my-homelab-configs
|
|||
./lab.sh up
|
||||
```
|
||||
|
||||
The script applies the OpenTofu stacks in order, refreshes Argo CD apps, waits
|
||||
for the local registry, builds the website and demos images when their source
|
||||
changed, pushes them to the registry, recreates pods only after a new image is
|
||||
built, and then applies the edge stack.
|
||||
The script detects the Pimox host at `192.168.100.80` in auto mode. When SSH,
|
||||
`qm`, and `vmbr0` are available, it applies `bootstrap/provisioning`, creates or
|
||||
reuses the Debian 13 arm64 template, creates or reuses one worker VM clone,
|
||||
discovers the guest IP through qemu-guest-agent, and passes that worker into the
|
||||
cluster layer. It then applies the remaining OpenTofu stacks, refreshes Argo CD
|
||||
apps, waits for the local registry, builds the website and demos images when
|
||||
their source changed, pushes them to the registry, recreates pods only after a
|
||||
new image is built, and applies the edge stack.
|
||||
|
||||
Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set
|
||||
`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline
|
||||
checks that the Pimox bridge already exists and refuses to edit Orange Pi host
|
||||
networking.
|
||||
|
||||
The website and demos images default to `linux/arm64` because both deployments
|
||||
are pinned to the Raspberry Pi worker. Override with `WEBSITE_IMAGE_PLATFORMS`
|
||||
|
|
@ -117,10 +127,11 @@ hostname.
|
|||
|
||||
## Adding Nodes
|
||||
|
||||
For Pimox on Orange Pi 5 Plus, use `bootstrap/provisioning` to create a Debian
|
||||
13 arm64 golden image first. The layer serves PXE, preseed, and guest-prep
|
||||
assets from the Debian homelab server, then the installed VM can be sealed and
|
||||
converted to a Pimox template. Details are in `bootstrap/provisioning/README.md`.
|
||||
For Pimox on Orange Pi 5 Plus, `./lab.sh up` can create the Debian 13 arm64
|
||||
template and worker VM clones automatically. Defaults are intentionally tied to
|
||||
the observed host: Pimox SSH host `192.168.100.80`, bridge `vmbr0`, template VMID
|
||||
`9000`, and worker VMIDs starting at `9010`. Details and override variables are
|
||||
in `bootstrap/provisioning/README.md`.
|
||||
|
||||
Add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file:
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,10 @@
|
|||
|
||||
This layer prepares a Debian server to PXE boot a Debian 13 arm64 VM and install a reusable worker-node golden image for Pimox on Orange Pi 5 Plus.
|
||||
|
||||
It is intentionally separate from `./lab.sh up`. Run it manually from the Debian homelab server only when you want to create or refresh the provisioning service.
|
||||
`./lab.sh up` drives this layer in auto mode when the Pimox host is reachable at
|
||||
`192.168.100.80`, `qm` is installed, and the existing bridge `vmbr0` is present.
|
||||
The automation creates VM definitions only; it validates the bridge and refuses
|
||||
to edit Orange Pi host networking.
|
||||
|
||||
## What It Installs
|
||||
|
||||
|
|
@ -16,7 +19,15 @@ It is intentionally separate from `./lab.sh up`. Run it manually from the Debian
|
|||
|
||||
## Apply From Debian
|
||||
|
||||
Find the LAN interface on the Debian server:
|
||||
The normal path is:
|
||||
|
||||
```bash
|
||||
cd ~/my-homelab-configs
|
||||
./lab.sh up
|
||||
```
|
||||
|
||||
For manual provisioning-only testing, find the LAN interface on the Debian
|
||||
server:
|
||||
|
||||
```bash
|
||||
ip -br addr
|
||||
|
|
@ -72,20 +83,21 @@ worker_nodes = {
|
|||
|
||||
Run the cluster layer from the Debian homelab server after the cloned VM is reachable over SSH.
|
||||
|
||||
## Optional Pimox Automation
|
||||
## Pimox Automation
|
||||
|
||||
Set `TF_VAR_pimox_template_builder_enabled=true` to have this layer SSH into the
|
||||
`./lab.sh up` sets `TF_VAR_pimox_template_builder_enabled=true` by default when
|
||||
Pimox is reachable. The layer SSHes into the
|
||||
Pimox host, create the template-build VM with `qm`, boot it from PXE, wait for
|
||||
the installed VM over SSH, run `/usr/local/sbin/homelab-prepare-template.sh`,
|
||||
power it off, switch boot order back to disk first, and run `qm template`.
|
||||
|
||||
The automation needs a predictable temporary IP for the installed VM. Use a DHCP
|
||||
reservation for the generated or configured MAC address, then set:
|
||||
The template sealing step discovers the installed VM IP through
|
||||
qemu-guest-agent, so a DHCP reservation is no longer required for the temporary
|
||||
template-build VM. If you still want to force a known address, set
|
||||
`TF_VAR_pimox_template_build_host`.
|
||||
|
||||
```bash
|
||||
export TF_VAR_pimox_template_builder_enabled=true
|
||||
export TF_VAR_pimox_host=192.168.100.91
|
||||
export TF_VAR_pimox_template_build_host=192.168.100.90
|
||||
LAB_PIMOX_PIPELINE=true ./lab.sh up
|
||||
```
|
||||
|
||||
Defaults match the observed Pimox VM shape: OVMF firmware, virtio networking,
|
||||
|
|
@ -93,3 +105,16 @@ virtio-scsi disk, `vmbr0`, `local` storage, 2 vCPU, and 2 GiB memory. Override
|
|||
`TF_VAR_pimox_template_scsi0`, `TF_VAR_pimox_template_efidisk0`,
|
||||
`TF_VAR_pimox_template_cores`, or `TF_VAR_pimox_template_memory` if the Orange
|
||||
Pi storage layout changes.
|
||||
|
||||
`./lab.sh up` also creates or reuses worker clones after the template exists. It
|
||||
defaults to one worker, VMID `9010`, names like `pimox-worker-01`, deterministic
|
||||
locally administered MAC addresses, and qemu-guest-agent IP discovery. Useful
|
||||
overrides:
|
||||
|
||||
```bash
|
||||
LAB_PIMOX_PIPELINE=false ./lab.sh up
|
||||
LAB_PIMOX_WORKER_COUNT=0 ./lab.sh up
|
||||
LAB_PIMOX_WORKER_COUNT=2 ./lab.sh up
|
||||
LAB_PIMOX_WORKER_BASE_VMID=9020 ./lab.sh up
|
||||
LAB_PIMOX_HOST=192.168.100.80 LAB_PIMOX_BRIDGE=vmbr0 ./lab.sh up
|
||||
```
|
||||
|
|
|
|||
|
|
@ -123,10 +123,12 @@ resource "null_resource" "pimox_template_vm_create" {
|
|||
pimox_host = var.pimox_host
|
||||
pimox_user = var.pimox_user
|
||||
ssh_key_path = var.pimox_ssh_key_path
|
||||
builder_version = "2"
|
||||
vmid = tostring(var.pimox_template_vmid)
|
||||
name = var.pimox_template_name
|
||||
cores = tostring(var.pimox_template_cores)
|
||||
memory = tostring(var.pimox_template_memory)
|
||||
bridge = var.pimox_template_bridge
|
||||
net0 = local.pimox_template_net0
|
||||
scsi0 = var.pimox_template_scsi0
|
||||
efidisk0 = var.pimox_template_efidisk0
|
||||
|
|
@ -153,8 +155,14 @@ if ! command -v qm >/dev/null 2>&1; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
if ! ip link show "${self.triggers.bridge}" >/dev/null 2>&1; then
|
||||
echo "Pimox bridge ${self.triggers.bridge} does not exist. Refusing to change Orange Pi networking." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if sudo qm status "$vmid" >/dev/null 2>&1; then
|
||||
if sudo qm config "$vmid" | grep -q '^template: 1$'; then
|
||||
sudo qm set "$vmid" --agent enabled=1
|
||||
exit 0
|
||||
fi
|
||||
if [ "$replace_existing" != "true" ]; then
|
||||
|
|
@ -184,10 +192,12 @@ sudo qm create "$vmid" \
|
|||
--ostype l26 \
|
||||
--scsihw virtio-scsi-pci \
|
||||
--sockets 1 \
|
||||
--vga virtio
|
||||
--vga virtio \
|
||||
--agent enabled=1
|
||||
|
||||
sudo qm set "$vmid" --efidisk0 "${self.triggers.efidisk0}"
|
||||
sudo qm set "$vmid" --scsi0 "${self.triggers.scsi0}"
|
||||
sudo qm set "$vmid" --agent enabled=1
|
||||
sudo qm start "$vmid"
|
||||
EOT
|
||||
]
|
||||
|
|
@ -200,35 +210,107 @@ resource "null_resource" "pimox_template_vm_seal" {
|
|||
depends_on = [null_resource.pimox_template_vm_create]
|
||||
|
||||
triggers = {
|
||||
host = var.pimox_template_build_host
|
||||
user = var.pimox_template_build_user
|
||||
ssh_key_path = var.pimox_template_build_ssh_key_path
|
||||
pimox_host = var.pimox_host
|
||||
pimox_user = var.pimox_user
|
||||
pimox_key_path = var.pimox_ssh_key_path
|
||||
guest_host = var.pimox_template_build_host
|
||||
guest_user = var.pimox_template_build_user
|
||||
guest_key_path = var.pimox_template_build_ssh_key_path
|
||||
seal_version = "2"
|
||||
timeout = var.pimox_template_build_timeout
|
||||
timeout_seconds = tostring(var.pimox_template_build_timeout_seconds)
|
||||
guest_ip_prefix = var.pimox_template_guest_ip_prefix
|
||||
vmid = tostring(var.pimox_template_vmid)
|
||||
}
|
||||
|
||||
connection {
|
||||
type = "ssh"
|
||||
user = self.triggers.user
|
||||
private_key = file(self.triggers.ssh_key_path)
|
||||
host = self.triggers.host
|
||||
timeout = self.triggers.timeout
|
||||
}
|
||||
provisioner "local-exec" {
|
||||
interpreter = ["/bin/bash", "-lc"]
|
||||
command = <<EOT
|
||||
set -euo pipefail
|
||||
|
||||
provisioner "remote-exec" {
|
||||
inline = [
|
||||
<<EOT
|
||||
set -eu
|
||||
pimox_host="${self.triggers.pimox_host}"
|
||||
pimox_user="${self.triggers.pimox_user}"
|
||||
pimox_key="${self.triggers.pimox_key_path}"
|
||||
guest_host="${self.triggers.guest_host}"
|
||||
guest_user="${self.triggers.guest_user}"
|
||||
guest_key="${self.triggers.guest_key_path}"
|
||||
timeout_seconds="${self.triggers.timeout_seconds}"
|
||||
guest_ip_prefix="${self.triggers.guest_ip_prefix}"
|
||||
vmid="${self.triggers.vmid}"
|
||||
|
||||
if [ -z "${self.triggers.host}" ]; then
|
||||
echo "pimox_template_build_host must point to the installed VM before template sealing can run" >&2
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "python3 is required to discover the Pimox guest IP from qemu-guest-agent" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sudo /usr/local/sbin/homelab-prepare-template.sh
|
||||
sudo nohup sh -c 'sleep 2; poweroff' >/dev/null 2>&1 &
|
||||
ssh_pimox() {
|
||||
ssh -i "$pimox_key" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "$pimox_user@$pimox_host" "$@"
|
||||
}
|
||||
|
||||
ssh_guest() {
|
||||
ssh -i "$guest_key" -o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "$guest_user@$guest_host" "$@"
|
||||
}
|
||||
|
||||
guest_ip_from_agent() {
|
||||
guest_json="$(ssh_pimox "sudo qm guest cmd '$vmid' network-get-interfaces" 2>/dev/null || true)"
|
||||
if [ -z "$guest_json" ]; then
|
||||
return 1
|
||||
fi
|
||||
GUEST_JSON="$guest_json" python3 - "$guest_ip_prefix" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
prefix = sys.argv[1]
|
||||
try:
|
||||
interfaces = json.loads(os.environ.get("GUEST_JSON", ""))
|
||||
except Exception:
|
||||
sys.exit(1)
|
||||
|
||||
for iface in interfaces or []:
|
||||
for address in iface.get("ip-addresses") or []:
|
||||
if address.get("ip-address-type") != "ipv4":
|
||||
continue
|
||||
ip = address.get("ip-address", "")
|
||||
if not ip or ip.startswith(("127.", "169.254.")):
|
||||
continue
|
||||
if prefix and not ip.startswith(prefix):
|
||||
continue
|
||||
print(ip)
|
||||
sys.exit(0)
|
||||
sys.exit(1)
|
||||
PY
|
||||
}
|
||||
|
||||
if ssh_pimox "sudo qm config '$vmid' | grep -q '^template: 1$'"; then
|
||||
ssh_pimox "sudo qm set '$vmid' --agent enabled=1"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
deadline=$((SECONDS + timeout_seconds))
|
||||
while (( SECONDS < deadline )); do
|
||||
if [ -z "$guest_host" ]; then
|
||||
guest_host="$(guest_ip_from_agent || true)"
|
||||
fi
|
||||
if [ -n "$guest_host" ] && ssh_guest "test -x /usr/local/sbin/homelab-prepare-template.sh"; then
|
||||
break
|
||||
fi
|
||||
sleep 15
|
||||
done
|
||||
|
||||
if [ -z "$guest_host" ]; then
|
||||
echo "Timed out waiting for VM $vmid to report a guest IP through qemu-guest-agent" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! ssh_guest "test -x /usr/local/sbin/homelab-prepare-template.sh"; then
|
||||
echo "Timed out waiting for SSH on template-build VM $vmid at $guest_host" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ssh_guest "sudo /usr/local/sbin/homelab-prepare-template.sh"
|
||||
ssh_guest "sudo nohup sh -c 'sleep 2; poweroff' >/dev/null 2>&1 &" || true
|
||||
EOT
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -241,6 +323,7 @@ resource "null_resource" "pimox_template_vm_finalize" {
|
|||
pimox_host = var.pimox_host
|
||||
pimox_user = var.pimox_user
|
||||
ssh_key_path = var.pimox_ssh_key_path
|
||||
finalizer_version = "2"
|
||||
vmid = tostring(var.pimox_template_vmid)
|
||||
}
|
||||
|
||||
|
|
@ -257,6 +340,11 @@ resource "null_resource" "pimox_template_vm_finalize" {
|
|||
set -eu
|
||||
|
||||
vmid="${self.triggers.vmid}"
|
||||
if sudo qm config "$vmid" | grep -q '^template: 1$'; then
|
||||
sudo qm set "$vmid" --agent enabled=1
|
||||
exit 0
|
||||
fi
|
||||
|
||||
elapsed=0
|
||||
while [ "$elapsed" -lt 600 ]; do
|
||||
if sudo qm status "$vmid" | grep -q 'status: stopped'; then
|
||||
|
|
@ -291,6 +379,7 @@ resource "null_resource" "provisioning_host" {
|
|||
http_port = tostring(var.http_port)
|
||||
config_hash = local.config_hash
|
||||
provisioning_layer = "1"
|
||||
provisioner_version = "2"
|
||||
}
|
||||
|
||||
connection {
|
||||
|
|
@ -362,7 +451,7 @@ install_missing_packages() {
|
|||
fi
|
||||
}
|
||||
|
||||
install_missing_packages ca-certificates curl dnsmasq nginx
|
||||
install_missing_packages ca-certificates curl dnsmasq nginx python3
|
||||
|
||||
sudo install -d -m 0755 \
|
||||
"$install_dir" \
|
||||
|
|
|
|||
|
|
@ -161,7 +161,7 @@ variable "pimox_template_builder_enabled" {
|
|||
|
||||
variable "pimox_host" {
|
||||
type = string
|
||||
default = "192.168.100.91"
|
||||
default = "192.168.100.80"
|
||||
}
|
||||
|
||||
variable "pimox_user" {
|
||||
|
|
@ -238,3 +238,13 @@ variable "pimox_template_build_timeout" {
|
|||
type = string
|
||||
default = "60m"
|
||||
}
|
||||
|
||||
variable "pimox_template_build_timeout_seconds" {
|
||||
type = number
|
||||
default = 3600
|
||||
}
|
||||
|
||||
variable "pimox_template_guest_ip_prefix" {
|
||||
type = string
|
||||
default = "192.168.100."
|
||||
}
|
||||
|
|
|
|||
346
lab.sh
346
lab.sh
|
|
@ -28,9 +28,352 @@ require_debian_server() {
|
|||
|
||||
run_tofu_stack() {
|
||||
local stack="$1"
|
||||
local -a apply_args=(-auto-approve)
|
||||
|
||||
if [[ "${stack}" == "bootstrap/cluster" && -n "${LAB_CLUSTER_VAR_FILE:-}" ]]; then
|
||||
apply_args+=("-var-file=${LAB_CLUSTER_VAR_FILE}")
|
||||
fi
|
||||
|
||||
tofu -chdir="${REPO_ROOT}/${stack}" init
|
||||
tofu -chdir="${REPO_ROOT}/${stack}" apply -auto-approve
|
||||
tofu -chdir="${REPO_ROOT}/${stack}" apply "${apply_args[@]}"
|
||||
}
|
||||
|
||||
truthy() {
|
||||
case "${1,,}" in
|
||||
1 | true | yes | on)
|
||||
return 0
|
||||
;;
|
||||
*)
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
disabled_value() {
|
||||
case "${1,,}" in
|
||||
0 | false | no | off | disabled)
|
||||
return 0
|
||||
;;
|
||||
*)
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
ensure_python3() {
|
||||
if command -v python3 >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y --no-install-recommends python3
|
||||
}
|
||||
|
||||
detect_route_interface() {
|
||||
local target="$1"
|
||||
|
||||
ip route get "${target}" 2>/dev/null | awk '
|
||||
{
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if ($i == "dev") {
|
||||
print $(i + 1)
|
||||
exit
|
||||
}
|
||||
}
|
||||
}
|
||||
'
|
||||
}
|
||||
|
||||
pimox_ssh() {
|
||||
local host="$1"
|
||||
local user="$2"
|
||||
local key_path="$3"
|
||||
|
||||
shift 3
|
||||
ssh -i "${key_path}" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${user}@${host}" "$@"
|
||||
}
|
||||
|
||||
pimox_guest_ipv4() {
|
||||
local guest_json
|
||||
local host="$1"
|
||||
local user="$2"
|
||||
local key_path="$3"
|
||||
local vmid="$4"
|
||||
local ip_prefix="$5"
|
||||
|
||||
guest_json="$(pimox_ssh "${host}" "${user}" "${key_path}" "sudo qm guest cmd '${vmid}' network-get-interfaces" 2>/dev/null || true)"
|
||||
if [[ -z "${guest_json}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
GUEST_JSON="${guest_json}" python3 - "${ip_prefix}" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
prefix = sys.argv[1]
|
||||
try:
|
||||
interfaces = json.loads(os.environ.get("GUEST_JSON", ""))
|
||||
except Exception:
|
||||
sys.exit(1)
|
||||
|
||||
for iface in interfaces or []:
|
||||
for address in iface.get("ip-addresses") or []:
|
||||
if address.get("ip-address-type") != "ipv4":
|
||||
continue
|
||||
ip = address.get("ip-address", "")
|
||||
if not ip or ip.startswith(("127.", "169.254.")):
|
||||
continue
|
||||
if prefix and not ip.startswith(prefix):
|
||||
continue
|
||||
print(ip)
|
||||
sys.exit(0)
|
||||
sys.exit(1)
|
||||
PY
|
||||
}
|
||||
|
||||
wait_for_pimox_guest_ssh() {
|
||||
local host="$1"
|
||||
local user="$2"
|
||||
local key_path="$3"
|
||||
local vmid="$4"
|
||||
local guest_user="$5"
|
||||
local guest_key_path="$6"
|
||||
local ip_prefix="$7"
|
||||
local timeout_seconds="$8"
|
||||
local deadline
|
||||
local guest_ip
|
||||
|
||||
deadline=$((SECONDS + timeout_seconds))
|
||||
while ((SECONDS < deadline)); do
|
||||
guest_ip="$(pimox_guest_ipv4 "${host}" "${user}" "${key_path}" "${vmid}" "${ip_prefix}" || true)"
|
||||
if [[ -n "${guest_ip}" ]] &&
|
||||
ssh -i "${guest_key_path}" -o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new "${guest_user}@${guest_ip}" true >/dev/null 2>&1; then
|
||||
printf '%s\n' "${guest_ip}"
|
||||
return 0
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
pimox_generated_mac() {
|
||||
local vmid="$1"
|
||||
|
||||
printf '02:68:10:%02x:%02x:%02x\n' \
|
||||
$(((vmid >> 16) & 255)) \
|
||||
$(((vmid >> 8) & 255)) \
|
||||
$((vmid & 255))
|
||||
}
|
||||
|
||||
ensure_pimox_worker_node() {
|
||||
local index="$1"
|
||||
local spec_file="$2"
|
||||
local pimox_host="$3"
|
||||
local pimox_user="$4"
|
||||
local pimox_key="$5"
|
||||
local template_vmid="$6"
|
||||
local bridge="$7"
|
||||
local worker_base_vmid="$8"
|
||||
local worker_name_prefix="$9"
|
||||
local worker_node_prefix="${10}"
|
||||
local worker_key_prefix="${11}"
|
||||
local worker_cores="${12}"
|
||||
local worker_memory="${13}"
|
||||
local worker_user="${14}"
|
||||
local worker_key_path="${15}"
|
||||
local ip_prefix="${16}"
|
||||
local timeout_seconds="${17}"
|
||||
local padded
|
||||
local vmid
|
||||
local worker_key
|
||||
local worker_name
|
||||
local node_name
|
||||
local mac
|
||||
local guest_ip
|
||||
|
||||
printf -v padded '%02d' "${index}"
|
||||
vmid=$((worker_base_vmid + index - 1))
|
||||
worker_key="${worker_key_prefix}${padded}"
|
||||
worker_name="${worker_name_prefix}-${padded}"
|
||||
node_name="${worker_node_prefix}-${padded}"
|
||||
mac="$(pimox_generated_mac "${vmid}")"
|
||||
|
||||
if pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo qm status '${vmid}' >/dev/null 2>&1"; then
|
||||
if pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo qm config '${vmid}' | grep -q '^template: 1$'"; then
|
||||
echo "VM ${vmid} exists as a template; refusing to reuse it as worker ${worker_name}." >&2
|
||||
exit 1
|
||||
fi
|
||||
pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo qm set '${vmid}' --agent enabled=1
|
||||
if sudo qm status '${vmid}' | grep -q 'status: stopped'; then sudo qm start '${vmid}'; fi"
|
||||
else
|
||||
pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "set -eu
|
||||
if ! ip link show '${bridge}' >/dev/null 2>&1; then
|
||||
echo 'Pimox bridge ${bridge} does not exist. Refusing to change Orange Pi networking.' >&2
|
||||
exit 1
|
||||
fi
|
||||
sudo qm clone '${template_vmid}' '${vmid}' --name '${worker_name}' --full 1
|
||||
sudo qm set '${vmid}' --agent enabled=1
|
||||
sudo qm set '${vmid}' --cores '${worker_cores}' --memory '${worker_memory}'
|
||||
sudo qm set '${vmid}' --net0 'virtio=${mac},bridge=${bridge}'
|
||||
sudo qm set '${vmid}' --boot 'order=scsi0;net0'
|
||||
sudo qm set '${vmid}' --onboot 1
|
||||
sudo qm start '${vmid}'"
|
||||
fi
|
||||
|
||||
if ! guest_ip="$(wait_for_pimox_guest_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "${vmid}" "${worker_user}" "${worker_key_path}" "${ip_prefix}" "${timeout_seconds}")"; then
|
||||
echo "Timed out waiting for worker VM ${vmid} (${worker_name}) to report a reachable guest IP." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' "${worker_key}" "${guest_ip}" "${worker_user}" "${node_name}" "${worker_key_path}" >>"${spec_file}"
|
||||
}
|
||||
|
||||
write_cluster_worker_var_file() {
|
||||
local spec_file="$1"
|
||||
local var_file="$2"
|
||||
|
||||
LAB_INCLUDE_RASPBERRY_WORKER="${LAB_INCLUDE_RASPBERRY_WORKER:-true}" \
|
||||
LAB_RASPBERRY_HOST="${LAB_RASPBERRY_HOST:-192.168.100.89}" \
|
||||
LAB_RASPBERRY_USER="${LAB_RASPBERRY_USER:-jv}" \
|
||||
LAB_RASPBERRY_NODE_NAME="${LAB_RASPBERRY_NODE_NAME:-raspberry}" \
|
||||
LAB_RASPBERRY_SSH_KEY_PATH="${LAB_RASPBERRY_SSH_KEY_PATH:-/home/jv/.ssh/id_ed25519}" \
|
||||
python3 - "${spec_file}" "${var_file}" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
spec_file, var_file = sys.argv[1:3]
|
||||
nodes = {}
|
||||
|
||||
if os.environ["LAB_INCLUDE_RASPBERRY_WORKER"].lower() not in {"0", "false", "no", "off", "disabled"}:
|
||||
nodes["raspberrypi"] = {
|
||||
"host": os.environ["LAB_RASPBERRY_HOST"],
|
||||
"user": os.environ["LAB_RASPBERRY_USER"],
|
||||
"node_name": os.environ["LAB_RASPBERRY_NODE_NAME"],
|
||||
"ssh_key_path": os.environ["LAB_RASPBERRY_SSH_KEY_PATH"],
|
||||
}
|
||||
|
||||
with open(spec_file, encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
line = line.rstrip("\n")
|
||||
if not line:
|
||||
continue
|
||||
key, host, user, node_name, ssh_key_path = line.split("\t")
|
||||
nodes[key] = {
|
||||
"host": host,
|
||||
"user": user,
|
||||
"node_name": node_name,
|
||||
"ssh_key_path": ssh_key_path,
|
||||
}
|
||||
|
||||
with open(var_file, "w", encoding="utf-8") as handle:
|
||||
json.dump({"worker_nodes": nodes}, handle, indent=2)
|
||||
handle.write("\n")
|
||||
PY
|
||||
}
|
||||
|
||||
run_pimox_pipeline() {
|
||||
local mode="${LAB_PIMOX_PIPELINE:-auto}"
|
||||
local pimox_host="${LAB_PIMOX_HOST:-${TF_VAR_pimox_host:-192.168.100.80}}"
|
||||
local pimox_user="${LAB_PIMOX_USER:-${TF_VAR_pimox_user:-jv}}"
|
||||
local pimox_key="${LAB_PIMOX_SSH_KEY_PATH:-${TF_VAR_pimox_ssh_key_path:-/home/jv/.ssh/id_ed25519}}"
|
||||
local bridge="${LAB_PIMOX_BRIDGE:-${TF_VAR_pimox_template_bridge:-vmbr0}}"
|
||||
local template_vmid="${LAB_PIMOX_TEMPLATE_VMID:-${TF_VAR_pimox_template_vmid:-9000}}"
|
||||
local template_name="${LAB_PIMOX_TEMPLATE_NAME:-${TF_VAR_pimox_template_name:-debian13-arm64-k8s-template}}"
|
||||
local provisioning_interface
|
||||
local worker_count="${LAB_PIMOX_WORKER_COUNT:-1}"
|
||||
local worker_base_vmid="${LAB_PIMOX_WORKER_BASE_VMID:-9010}"
|
||||
local worker_name_prefix="${LAB_PIMOX_WORKER_NAME_PREFIX:-pimox-worker}"
|
||||
local worker_node_prefix="${LAB_PIMOX_WORKER_NODE_PREFIX:-pimox-worker}"
|
||||
local worker_key_prefix="${LAB_PIMOX_WORKER_KEY_PREFIX:-pimox}"
|
||||
local worker_cores="${LAB_PIMOX_WORKER_CORES:-2}"
|
||||
local worker_memory="${LAB_PIMOX_WORKER_MEMORY:-2048}"
|
||||
local worker_user="${LAB_PIMOX_WORKER_USER:-jv}"
|
||||
local worker_key_path="${LAB_PIMOX_WORKER_SSH_KEY_PATH:-/home/jv/.ssh/id_ed25519}"
|
||||
local ip_prefix="${LAB_PIMOX_GUEST_IP_PREFIX:-192.168.100.}"
|
||||
local timeout_seconds="${LAB_PIMOX_GUEST_TIMEOUT_SECONDS:-3600}"
|
||||
local spec_file="${REPO_ROOT}/.lab/pimox-workers.tsv"
|
||||
local var_file="${REPO_ROOT}/.lab/cluster-workers.auto.tfvars.json"
|
||||
local index
|
||||
|
||||
if disabled_value "${mode}"; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ! [[ "${worker_count}" =~ ^[0-9]+$ ]]; then
|
||||
echo "LAB_PIMOX_WORKER_COUNT must be a non-negative integer." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "command -v qm >/dev/null 2>&1 && ip link show '${bridge}' >/dev/null 2>&1"; then
|
||||
if [[ "${mode}" == "auto" ]]; then
|
||||
echo "Skipping Pimox automation because ${pimox_user}@${pimox_host} with bridge ${bridge} is not ready."
|
||||
return 0
|
||||
fi
|
||||
echo "Pimox automation requested, but ${pimox_user}@${pimox_host} is not reachable or bridge ${bridge} is missing." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ensure_python3
|
||||
provisioning_interface="${TF_VAR_provisioning_interface:-${LAB_PROVISIONING_INTERFACE:-$(detect_route_interface "${pimox_host}")}}"
|
||||
if [[ -z "${provisioning_interface}" ]]; then
|
||||
echo "Could not detect the Debian interface used to reach ${pimox_host}; set LAB_PROVISIONING_INTERFACE." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export TF_VAR_provisioning_interface="${provisioning_interface}"
|
||||
export TF_VAR_pimox_host="${pimox_host}"
|
||||
export TF_VAR_pimox_user="${pimox_user}"
|
||||
export TF_VAR_pimox_ssh_key_path="${pimox_key}"
|
||||
export TF_VAR_pimox_template_bridge="${bridge}"
|
||||
export TF_VAR_pimox_template_vmid="${template_vmid}"
|
||||
export TF_VAR_pimox_template_name="${template_name}"
|
||||
export TF_VAR_pimox_template_builder_enabled="${TF_VAR_pimox_template_builder_enabled:-true}"
|
||||
export TF_VAR_pimox_template_build_ssh_key_path="${TF_VAR_pimox_template_build_ssh_key_path:-${worker_key_path}}"
|
||||
export TF_VAR_pimox_template_build_user="${TF_VAR_pimox_template_build_user:-${worker_user}}"
|
||||
export TF_VAR_pimox_template_guest_ip_prefix="${TF_VAR_pimox_template_guest_ip_prefix:-${ip_prefix}}"
|
||||
export TF_VAR_pimox_template_build_timeout_seconds="${TF_VAR_pimox_template_build_timeout_seconds:-${timeout_seconds}}"
|
||||
|
||||
echo "Preparing Pimox provisioning and Debian worker template on ${pimox_host} without changing Orange Pi host networking..."
|
||||
run_tofu_stack "bootstrap/provisioning"
|
||||
|
||||
if ((worker_count == 0)); then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ! pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo qm config '${template_vmid}' | grep -q '^template: 1$'"; then
|
||||
echo "Template VM ${template_vmid} is not available as a Pimox template after provisioning." >&2
|
||||
exit 1
|
||||
fi
|
||||
pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo qm set '${template_vmid}' --agent enabled=1"
|
||||
|
||||
mkdir -p "${REPO_ROOT}/.lab"
|
||||
: >"${spec_file}"
|
||||
for ((index = 1; index <= worker_count; index++)); do
|
||||
ensure_pimox_worker_node \
|
||||
"${index}" \
|
||||
"${spec_file}" \
|
||||
"${pimox_host}" \
|
||||
"${pimox_user}" \
|
||||
"${pimox_key}" \
|
||||
"${template_vmid}" \
|
||||
"${bridge}" \
|
||||
"${worker_base_vmid}" \
|
||||
"${worker_name_prefix}" \
|
||||
"${worker_node_prefix}" \
|
||||
"${worker_key_prefix}" \
|
||||
"${worker_cores}" \
|
||||
"${worker_memory}" \
|
||||
"${worker_user}" \
|
||||
"${worker_key_path}" \
|
||||
"${ip_prefix}" \
|
||||
"${timeout_seconds}"
|
||||
done
|
||||
|
||||
write_cluster_worker_var_file "${spec_file}" "${var_file}"
|
||||
export LAB_CLUSTER_VAR_FILE="${var_file}"
|
||||
}
|
||||
|
||||
cleanup_calico_links() {
|
||||
|
|
@ -668,6 +1011,7 @@ up() {
|
|||
|
||||
echo "Deploying the homelab infrastructure..."
|
||||
|
||||
run_pimox_pipeline
|
||||
run_tofu_stack "bootstrap/cluster"
|
||||
run_tofu_stack "bootstrap/platform"
|
||||
apply_gitea_bootstrap_manifests
|
||||
|
|
|
|||
Loading…
Reference in New Issue