diff --git a/README.md b/README.md index 07bcb59..a6b69e4 100644 --- a/README.md +++ b/README.md @@ -2,324 +2,9 @@ -This repo bootstraps a hybrid kubeadm cluster and then hands app delivery to -Argo CD. +This repo bootstraps a hybrid kubeadm cluster and then hands app delivery to Argo CD. -## Architecture - -The lab is intentionally small but production-shaped: - -- a Debian amd64 host runs the kubeadm control plane and local deployment tools -- a Raspberry Pi arm64 node runs selected workloads -- the Debian host also runs the always-on Gitea Docker service outside - Kubernetes -- the Debian host keeps a bare GitOps mirror under - `/home/jv/git-server/my-homelab-configs.git` -- a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker - templates -- OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge -- Argo CD continuously reconciles Kubernetes manifests from this repo -- a local registry stores the website and demos images built for the worker - architecture -- the website image keeps Apache as the public server, uses PHP-FPM with - OPcache for dynamic endpoints, and pre-renders read-heavy pages to static - HTML during the image build -- SOPS with age is the committed secret-management path for future encrypted - Kubernetes secrets -- local Jeannie/RAG eval cases exercise troubleshooting explanations and - retrieval grounding without touching live infrastructure -- an OCI jump box provides the public edge path back into the homelab over - Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream - keepalive - -## Inventory - -The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, -Tailscale IPs, public hostnames, ports, storage paths, and service ownership -there before copying values into scripts, OpenTofu variables, runbooks, or -service docs. Secrets and tokens do not belong in that file. - -`./jeannie` reads `homelab.yml` at startup and exports matching OpenTofu -variables for the edge, provisioning, cluster, platform, and apps stacks. This -keeps values such as the Debian LAN IP, Debian Tailscale IP, RPi LAN IP, OCI IP, -Traefik MetalLB IP, and domain name from drifting across scripts and templates. -`scripts/render-docs` also renders [docs/service-catalog.md](docs/service-catalog.md) -and the static service catalog page at -`apps/demos-static/public/homelab-catalog/index.html` from the same inventory. - -Run `./jeannie up` and `./jeannie nuke` only from the Debian homelab server. The -script intentionally refuses to run from non-Debian machines so a laptop cannot -accidentally modify the cluster. - -## Flow - -1. `bootstrap/provisioning` - - prepares a Debian server as a PXE and preseed service for arm64 VMs - - serves Debian 13 arm64 netboot assets through TFTP and HTTP - - exposes a PXE rescue menu and HTTP rescue page for manual repair paths - - creates a golden image install path with Kubernetes, containerd, - qemu-guest-agent, cloud-init, and storage client packages ready - - is driven by `./jeannie up` when Pimox is reachable, without changing - Orange Pi host networking - -2. `bootstrap/cluster` - - creates the kubeadm control plane on the Debian amd64 node - - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes - - configures Calico-compatible pod CIDR - - configures containerd to pull from the in-cluster NodePort registry - - creates retained host directories under `/data/openebs/local` - -3. `bootstrap/platform` - - installs a minimal Calico deployment through the Tigera operator - - installs NodeLocal DNSCache for node-local DNS query caching - - installs MetalLB for LAN `LoadBalancer` services - - installs Traefik as the single Kubernetes ingress entry point - - installs OpenEBS - - creates `openebs-hostpath-retain` - - installs Argo CD - - installs Kyverno with audit-first baseline Pod Security policies and - homelab image signature/SBOM admission controls - - registers the private GitOps repo without storing the SSH private key in - Terraform state - -4. `bootstrap/apps` - - registers Argo CD Applications from the `applications` map - - passes the website image produced by the build step into Argo CD as a - Kustomize image override - - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM - listens on loopback inside the container, OPcache is enabled, and - pre-rendered HTML handles the read-heavy pages - - default apps are `container-registry`, `website-production`, - `demos-static`, the `heimdall` route bundle, `n8n`, and - `supply-chain-policy` - -5. `bootstrap/edge` - - connects to the OCI jump box - - uploads nginx, HAProxy, Varnish, and Squid configs - - obtains and renews Let's Encrypt certificates for the configured hostname - - runs the edge cache/proxy chain with Docker Compose - - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream - keepalive, and cache headers for the public website path - -## Prerequisites - -On the Debian host: - -- OpenTofu -- Docker with Buildx -- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes - minor, currently `v1.36` -- SSH access to worker nodes -- SSH access to the OCI edge host -- Go toolchain for Jeannie core helpers; `./jeannie up` installs - `golang-go` on the Debian host if it is missing -- enough persistent storage for `/data/openebs/local` and Docker's data root - -After a clean Debian reinstall, bootstrap the host package set and desktop -configuration first: - -```bash -sudo apt-get update -sudo apt-get install -y git ansible sudo -git clone /home/jv/my-homelab-configs -cd /home/jv/my-homelab-configs -ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml -``` - -If you kept the private app-config archive generated from the old Debian host, -pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. -Details are in `bootstrap/host/README.md`. - -The default kubeconfig path is `/home/jv/.kube/config`. Override it with -`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. - -## Implementation Language Policy - -Jeannie keeps `./jeannie` as the stable command entrypoint. New durable -read-only orchestration, report rendering, inventory parsing, status/doctor -logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because -it ships as one small binary and is easier to test than Bash. Bash remains for -thin compatibility wrappers and remote shell glue. Python remains acceptable for -AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the -clear fit. New repo features should choose the smallest language that fits the -task instead of defaulting to Bash. - -## Version Discipline - -The lab should converge on one Kubernetes minor across the API server and every -kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers -match the Debian control plane and Raspberry Pi worker instead of staying on an -older template. Avoid treating a three-minor kubelet skew as normal steady -state: it is only a temporary compatibility window and blocks the next control -plane upgrade. - -Check node versions from the Debian host: - -```bash -./jeannie doctor-versions -``` - -The check prints the API server, kubelet versions, kubelet minors, and containerd -runtimes. It exits nonzero when kubelet minors differ from the API server minor -or when containerd major versions are mixed. The Gitea Actions auto path runs -this check before app deploys; kubelet minor drift switches the job to -`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced -from the pinned image. For Pimox workers, prefer rebuilding from the corrected -golden template and rejoining the worker over ad hoc live package upgrades. - -## AI Evals - -The repo includes a local eval harness for Jeannie explanations and homelab RAG -grounding. It uses fixtures and allowlisted read-only commands, so it is safe to -run before changing the lab: - -```bash -./jeannie ai-evals run -``` - -Use `./jeannie ai-evals list` to see the current cases. Add new cases -under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior -should become repeatable. - -## Prompt Injection Lab - -The defensive security training path includes local prompt-injection fixtures -for hostile runbooks, tool output, dashboard annotations, and command arguments. -Run it without touching live infrastructure: - -```bash -./jeannie prompt-injection-lab run -``` - -Cases live under `security/prompt-injection-lab` and are meant to become -regression tests for future Jeannie AI features. - -## Incident Commander - -The repo also includes a read-only incident commander trainer. It classifies -pasted failure text, prints a likely hypothesis, links the right runbook, and -lists the next read-only Jeannie commands: - -```bash -./jeannie incident replay -./jeannie incident triage --text "502 Bad Gateway from nginx" -``` - -Incident fixtures live under `infra/incident-commander`. Use them to turn -repeat outages into repeatable triage behavior before adding any self-healing. - -## Safety Cases - -Before high-blast-radius changes, generate a safety case from the Git diff: - -```bash -./jeannie safety-case --since HEAD~1 -``` - -It summarizes risk, blast radius, required validation, rollback, and forbidden -actions. The implementation lives under `infra/safety-cases` and reuses the -same impact rules as `./jeannie impact`. - -## Agent Sandbox - -Future Jeannie AI actions should be checked before execution. The local sandbox -policy starts with a read-only command classifier: - -```bash -./jeannie agent-sandbox check "./jeannie status" -./jeannie agent-sandbox check "./jeannie nuke" -``` - -Policy is stored in `infra/agent-sandbox/policy.tsv`. - -## AI Workload Scheduling - -For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox -workers, and the RPi based on CPU, memory, architecture, tailnet access, and -control-plane policy: - -```bash -./jeannie ai-scheduler recommend small-rag -./jeannie ai-scheduler simulate -``` - -The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a -real Kubernetes controller. - -## Memory With Provenance - -Jeannie can return cited retrieval context and check whether the local RAG index -is stale: - -```bash -./jeannie ask --citations "how do I recover the cluster?" -./jeannie ai-memory check -``` - -This keeps local AI answers grounded in repo files and runbooks. - -## Red/Blue Loop - -Safe red-team/blue-team checks are grouped under one command: - -```bash -./jeannie red-blue plan -./jeannie red-blue run --local -``` - -Local mode runs deterministic prompt-injection and eval checks. Live scans are -listed but gated so they can run from the Debian host when intended. - -## Canary Promotion - -The promotion helper defines release gates before production changes: - -```bash -./jeannie promote plan -./jeannie promote validate -./jeannie promote rollback website-production -``` - -The current version is intentionally conservative: it validates local gates and -prints rollback steps while dev namespace automation remains explicit future -work. - -## Guarded Self-Healing - -Jeannie can turn status findings into an experimental self-heal plan: - -```bash -./jeannie status --heal-plan -./jeannie status --heal -./jeannie heal apply --yes -``` - -Normal `status` remains read-only. The heal planner maps known findings to -guarded actions, chooses one highest-impact auto-eligible finding, checks the -proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that -one action. Re-run `./jeannie status` after every heal cycle so Jeannie -can reassess before touching the next issue. If a specific Pimox worker is -NotReady, heal targets that worker with `./jeannie workers restart N` -instead of repeating a broad cluster start. After an action runs, Jeannie waits -for the target condition to clear and reports `healed` or `still failing`. -Failed heal actions are placed on a short cooldown so Jeannie does not loop on -the same VM restart; the next plan escalates to diagnostics instead. -Destructive actions remain blocked. - -## Model Behavior Observatory - -Stable prompts track local model behavior, latency, refusal behavior, and -tool-correctness signals over time: - -```bash -./jeannie model-observe prompts -./jeannie model-observe run --offline -./jeannie model-observe report -``` - -Use `--live` only when Ollama is available. - -## Deploying +## Quick Start From the Debian server: @@ -328,1161 +13,21 @@ cd ~/my-homelab-configs ./jeannie up ``` -The script first deploys external Gitea to the Debian host with Docker Compose -under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays -outside the Kubernetes rebuild blast radius. It then detects the -Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are -available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 -arm64 template, creates or reuses one worker VM clone, discovers the guest IP -through qemu-guest-agent, and passes that worker into the cluster layer. It then -applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the -local registry, builds the website and demos images when their source changed, -pushes them to the registry, recreates pods only after a new image is built, and -applies the edge stack. +## Documentation Hub -Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set -`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline -keeps the template on its configured `local` storage, creates new worker VM -clones on `opi5_ssd` by default when that Pimox storage is available, -checks that the Pimox bridge already exists, refuses `local` as worker clone -storage, and refuses to edit Orange Pi host networking. Keep durable data on -the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks -as rebuildable worker capacity. +The detailed technical specifications have been moved to the `/docs` directory: -`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns -worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set -`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker -must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID -`9010` and VMID `9011`. +- [Architecture & Infrastructure](docs/architecture.md) - Cluster layout, Networking, and Edge path. +- [Deployment Guide](docs/deployment.md) - Prerequisites, bootstrap flow, and validation. +- [AI Operations](docs/ai-ops.md) - Jeannie AI tools, Evals, and Safety cases. +- [Application Ops](docs/app-ops.md) - App delivery, Gitea, and Website management. +- [Service Catalog](docs/service-catalog.md) - Live inventory of all hosts and services. -Pimox workers can join the tailnet during cluster worker bootstrap. Keep the -ephemeral or pre-authorized auth key outside Git, either in -`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in -`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: +## Inventory -```bash -LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up -``` +The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, Tailscale IPs, public hostnames, ports, storage paths, and service ownership there. -For existing workers, prefer the Jeannie wrapper because it reads the generated -worker keys and exports the correct OpenTofu variables: +`./jeannie` reads `homelab.yml` at startup and exports matching OpenTofu variables. +`scripts/render-docs` renders the documentation and the static service catalog page. -```bash -./jeannie workers tailnet -``` - -When applying `bootstrap/cluster` directly, use the OpenTofu variable name -instead of the Jeannie `LAB_` wrapper: - -```bash -TF_VAR_worker_tailscale_enabled=true \ -tofu -chdir=bootstrap/cluster apply \ - -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" -``` - -Worker Tailscale defaults to `--accept-routes=false` because these VMs already -live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can -steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT -from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox -workers can reach tailnet-only services. Set -`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume -tailnet-advertised routes that do not overlap the local LAN. Set -`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through -`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox -worker tailnet egress against -`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and -`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. - -OpenWrt firewall VM automation is available as a standalone command because it -attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` -already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM -SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, -imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores -the VM disk on the selected non-local Pimox storage. It leaves the VM stopped -and not enabled for host boot by default. It does not use the Debian Kubernetes -golden-node template for OpenWrt. - -The website and demos images default to `linux/amd64,linux/arm64` because both -deployments can schedule on either the Debian host or ARM workers. Override with -`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally -restricting node placement. - -Build metadata is written under `.lab/` so repeat runs can skip the website -or demos image build when the source hash, platform, image reference, and -registry manifest still match. - -The website build is intentionally performed on the Debian homelab path and the -Gitea Actions runner, not on a laptop. The current image uses -`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables -OPcache for immutable image deploys, and runs -`apps/website/render-static-pages.sh` during the image build. That render step -produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and -`/homelab-tree.php` in the static languages. Runtime PHP remains for write and -translation endpoints such as `save_idea.php`, `visitor_ideas.php`, -`translate.php`, and `save_lang.php`. - -Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the -service is already managed manually. The default Gitea target is -`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, -and SSH port `32222`. - -Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` -runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker -root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, -Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory -or host storage/networking: - -```bash -./jeannie preflight -``` - -If the Debian Docker root check fails after a reinstall, repair it before -running the full pipeline: - -```bash -./jeannie fix-debian-docker-root -``` - -When `./jeannie up` sees an existing tracked kubeadm control plane but -the Kubernetes API is down, it starts the saved runtime before applying cluster -changes. Use `./jeannie rebuild-cluster` only when you intentionally -want to destroy and recreate the cluster. - -## Validation - -Useful checks after a rebuild: - -```bash -export KUBECONFIG=/home/jv/.kube/config - -kubectl get nodes -kubectl top nodes -kubectl -n argocd get applications -kubectl -n container-registry get pods -kubectl -n website-production get pods -o wide -kubectl -n demos-static get pods -o wide - -ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' -ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' - -docker info --format '{{.DockerRootDir}}' -df -h / /data /data/openebs/local /data/docker -``` - -The website should be reached through the configured public hostname, not the raw -OCI IP address, because the Let's Encrypt certificate is issued for the -hostname. - -Run a read-only health snapshot from the Debian server with: - -```bash -./jeannie help -./jeannie status -./jeannie capacity -./jeannie capacity-advisor -./jeannie capacity-limits -./jeannie recover-plan -./jeannie recover-power --dry-run -./jeannie impact --since HEAD~1 -./jeannie review-last-change -./jeannie scorecard -./jeannie explain scorecard -./jeannie gitops-status -./jeannie cert-check -./jeannie grafana-dashboards list -./jeannie release-snapshot -./jeannie backup-status -./jeannie synthetic-checks -./jeannie resource-budget -./jeannie control-plane status -./jeannie artifact-cache status -./jeannie blockchain-devnet status -./jeannie blockchain-test -./jeannie blockchain-wallet instructions -./jeannie golden-ledger check -./jeannie route-inventory -./jeannie workers list -./jeannie change-journal list -./jeannie map -``` - -`help` prints the grouped command reference in pipeline order, from inventory -and bootstrap through cluster lifecycle, reports, observability, recovery, -access, apps, security, AI, and destructive commands. - -`status` uses the reusable report UI and defaults to showing only failures and -warnings so the terminal stays readable. Use `./jeannie status --all` -to show every check, `--details` for row details, `--json` for structured -output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ -Tailscale/HTTP dump. The cascade includes "what broke" signals such as -deployment readiness, pod restarts, node pressure, disk pressure, Traefik -5xx/404 log evidence, and recent Gitea errors. - -High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node -readiness, pod restart storms, unavailable deployments, node/PVC storage -pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape -coverage. Keep this set small so alerts remain actionable. - -Application egress is controlled with Kubernetes NetworkPolicies where the lab -owns the workload. The website namespace defaults to denied egress and allows -only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is -isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS -for controlled practice targets. - -`capacity` is the placement report for the hardware you already have. By -default it uses the compact report renderer and shows summarized Debian, -Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use -`./jeannie capacity --details`, `--json`, `--only problems`, or -`--verbose` for the older full capacity dump. - -`capacity-advisor` interprets capacity, resource-budget, and control-plane -placement signals to recommend whether to add Pimox VMs, taint Debian, or fix -resource definitions first. - -`capacity-limits` prints suggested Kubernetes `resources.requests` and -`resources.limits` YAML snippets for containers missing them. It uses -`infra/resource-recommendations.yml` as a conservative starting profile and does -not edit manifests. - -`recover-plan` prints the disaster recovery order and lightweight prerequisite -checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and -edge verification. - -`recover-power` runs the post-outage recovery sequence in dependency order: -Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and -edge/public checks. Use `recover-power --dry-run` to preview the sequence. - -`impact` explains what a change may touch before you apply it. Use -`./jeannie impact --since HEAD~1` for the last commit or pass paths -like `bootstrap/edge` and `apps/website`. - -`review-last-change` summarizes the last commit, runs impact analysis for the -touched files, and prints validation plus rollback hints. - -`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster -health, edge health, capacity pressure, security posture, public certificates, -inventory, and docs freshness into a compact pass/warn/fail report. - -`explain` is the read-only follow-up for warnings, failures, and corrective -output. It can run safe report commands itself, such as -`./jeannie explain scorecard`, `status`, `capacity`, or -`resource-budget`, and it can explain captured output with -`./jeannie explain output < saved-output.txt`. It lists what each -finding means, diagnostic commands, whether a Jeannie fix command already -exists, and risk/auto-fix safety. - -Report-oriented commands share the reusable UI module in `scripts/report-ui` -and renderer in `scripts/report-render`. Commands emit status TSV rows, and the -renderer prints compact grouped summaries by default, with details or JSON -available for future TUI/web output. Rows may include separate `fix`, `graph`, -and `query` fields so a warning can point to both the next `jeannie` command and -the matching Grafana panel or Explore query. - -`gitops-status` prints a focused Argo CD view: application sync and health, -out-of-sync or degraded apps, repository secret presence, recent Argo CD events, -and recent repo-server/application-controller errors. - -`grafana-dashboards` manages repo-provisioned Grafana dashboards under -`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions -them through the platform stack; `./jeannie grafana-dashboards apply` -rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional -blockchain devnet dashboard uses metrics from -`infra/blockchain-devnet/metrics/exporter.py`. - -`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea -HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. - -`release-snapshot` writes a timestamped pre-change report under the homelab -state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, -public URL status, and pointers to the latest Gitea/OpenTofu backups. - -`backup-status` reports freshness for Gitea backups, OpenTofu state backups, -restore drill reports, and repo-managed Pi-hole restore inputs. The main -`status` cascade includes this as a warning signal. - -`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, -Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The -registry push/pull smoke test is opt-in with -`LAB_SYNTHETIC_REGISTRY_PUSH=true`. - -`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` -against Kubernetes pod requests/limits and Debian host disk/memory signals. Use -it before adding heavier workloads so the HP laptop remains usable. By default -it summarizes missing requests/limits by namespace; use -`./jeannie resource-budget --details` for every pod/container. - -`control-plane` shows or manages the Debian Kubernetes control-plane scheduling -taint. After Pimox workers are stable, use `./jeannie control-plane taint` -to keep normal app/data-plane pods off the Debian host while control-plane and -DaemonSet workloads continue using their normal tolerations. - -`artifact-cache` manages an optional Debian-host cache stack in -`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through -cache. Use `artifact-cache instructions` for client configuration snippets. - -`blockchain-devnet` manages an optional local Ethereum Anvil devnet under -`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability -learning only; never use real seed phrases or mainnet keys. - -`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts -with a simple `Counter` contract and ownership tests so there is a known-good -baseline before practicing vulnerable contracts. - -`blockchain-wallet` provides disposable dev-wallet generation, address -derivation, and message-signing practice. It is documented in -`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. - -`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the -reviewable record of template VMID, storage, OS release, Kubernetes pins, -runtime versions, and build metadata for Pimox worker golden images. - -`route-inventory` reports Kubernetes ingress hosts, paths, backend services, -TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and -missing monitors are easier to spot. - -`workers` provides named Kubernetes/Pimox worker lifecycle operations: -`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and -`rebalance`. -Use `recreate-plan ` before replacing one broken Pimox worker so the -cluster drain, VM stop, and `./jeannie up` path stay ordered. - -`change-journal` lists local journal entries written before risky commands such -as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, -`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live -under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, -dirty file count, and command details. - -`map` prints a dependency map from the canonical inventory, covering public -entry, GitOps automation, cluster foundation, DNS, observability, and security -learning paths. `./jeannie map --dot` emits Graphviz DOT. - -Focused doctor commands run narrower read-only checks. They use the same -problems-first UI as `status`; add `--all`, `--details`, `--json`, or -`--verbose` when you need more context: - -```bash -./jeannie doctor-edge -./jeannie doctor-gitea -./jeannie doctor-rpi -./jeannie doctor-cluster -``` - -`doctor-cluster` also checks whether Pimox workers have Tailscale installed, -`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet -probe service. - -## RPi Services - -The Raspberry Pi runs lightweight always-on services outside Kubernetes from -`infra/rpi-services`: - -- Pi-hole on DNS port `53` and web port `8081` -- Unbound as Pi-hole's first upstream resolver -- Uptime Kuma on port `3001` - -Deploy or refresh them from the Debian server with: - -```bash -./jeannie rpi-services -``` - -`./jeannie up` also runs this step unless -`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is -expected under `/nvme-storage/docker` when that path is an active writable mount point. If -the NVMe is missing or resets, it rewrites Docker's `data-root` to -`/var/lib/docker` and restarts Docker so DNS can still come up from the -SD/root filesystem. - -Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can -continue resolving if the Unbound container stops. Repo-managed Pi-hole state -lives in `infra/rpi-services/adlists.txt`, -`infra/rpi-services/local-dns-records.txt`, -`infra/rpi-services/cname-records.txt`, and -`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without -deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into -a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` -is not provided, the bootstrap generates one and keeps it in -`/opt/homelab-rpi-services/.env`. -Uptime Kuma monitor seeds live in -`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first -Kuma user exists. -`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, -direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and -whether Docker is currently using `/nvme-storage/docker` or has fallen back to -`/var/lib/docker`. - -## Adding Nodes - -For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 -template and worker VM clones automatically when the Orange Pi storage path is -healthy. Defaults are intentionally tied to the observed host: Pimox SSH host -`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two -8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, -and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye -on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that -host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` -entries so native `qm` CPU topology is the only `-smp` argument. Set -`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe -that was previously attached to the Orange Pi now belongs to the HP Debian -laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox -datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and -override variables are in -`bootstrap/provisioning/README.md`. - -Worker indexes are stable. Index `1` maps to VMID `9010`, node name -`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and -so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while -allowing higher indexes to be automated. - -Worker networking defaults to static because the ISP modem DHCP server does not -answer Pimox worker VM DHCP requests behind `vmbr0`. Set -`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM -traffic. Static mode uses -`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, -`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and -`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. -For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions -for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` -otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before -adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` -only when the Pimox host firewall already permits bridged worker IP traffic. -Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under -`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. -Worker NIC MAC addresses default to the VMID-derived deterministic format used -by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` -only when you explicitly want Pimox to generate the MAC. -Worker creation fails faster than template creation: static guest-network -configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, -worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, -and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. -Static guest-network validation skips the extra LAN ICMP probe by default -because the ISP modem may not answer worker VM pings. Set -`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host -should be required during qemu-guest-agent network setup. After -qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by -default and lets SSH be the reachability check. Set -`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. -`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. -Before `./jeannie apps` changes any application resources, Jeannie requires -every generated Pimox worker to be `Ready` and expose at least two Kubernetes -CPUs. Override the threshold with `LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set -`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately -non-Pimox deployment. - -Long-running `./jeannie up` steps keep compact output by default, but -the active step line refreshes with elapsed time and the latest log line. Set -`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or -`JEANNIE_VERBOSE=true` to stream full command output. -Independent host services run through the Go DAG runner during `up`: Gitea -deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks -can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential -execution while troubleshooting, or raise it above the default `4` if the lab -can absorb more concurrent work. - -Run a full cluster rebuild from the Debian server with: - -```bash -./jeannie rebuild-cluster -``` - -That path preserves external Gitea, rebuilds the Pimox template -with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and -8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM -does not support `qm --affinity`; Jeannie instead starts the generated KVM -command under CPU set `4-7` and verifies every worker reports two guest CPUs -before Kubernetes workloads proceed. -The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` -does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. - -To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, -PV data, or VM disks, run: - -```bash -./jeannie stop-cluster -``` - -This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when -available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully -shuts down automated Pimox worker VMs. It intentionally leaves the host -`containerd`/Docker stack running so external Docker Compose services such as -Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove -state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want -to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` -to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a -hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, -`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to -select VM workers. Resume the runtime with: - -```bash -./jeannie start-cluster -``` - -Kubernetes worker generation defaults to automated Pimox workers only. The -Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` -is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless -`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, -add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: - -```hcl -worker_nodes = { - raspberrypi = { - host = "192.168.100.89" - user = "jv" - node_name = "raspberry" - ssh_key_path = "/home/jv/.ssh/id_ed25519" - } -} -``` - -Stateful apps currently pin retained local PVs to the `debian` node. Move or -duplicate those PV manifests when you want storage on another node. - -## Workload Placement - -`bootstrap/cluster` labels nodes with homelab placement metadata: - -- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` - shows `worker` instead of `` in the ROLES column -- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and - `homelab.dev/workload-class=control-plane` on the Debian control plane -- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and - `homelab.dev/workload-class=edge` on the Raspberry Pi worker -- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and - `homelab.dev/workload-class=platform` on automated Pimox worker clones when - those workers are enabled - -Override `control_plane_node_labels`, `worker_node_labels`, -`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when -the physical layout changes. The current website, demos, and registry manifests -are not moved automatically because the public NodePort path and retained -OpenEBS hostpath PVs are node-local. Move workloads only after their storage and -edge path are ready on the target node. Gitea is outside Kubernetes and is moved -by changing the Raspberry Pi Docker install target instead. - -The stateless platform controllers are pinned to Pimox worker nodes through -`homelab.dev/workload-class=platform` and include hostname topology spread plus -preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and -kube-state-metrics scheduling does not collapse onto the first worker that joins. -PVC-backed monitoring StatefulSets are intentionally treated separately because -their retained OpenEBS hostpath volumes are node-local. Run -`./jeannie move-prometheus-stack-workers` from the Debian host to label existing -worker nodes, destroy only the existing `prometheus-stack` Helm release, delete -its retained PVC/PV objects, and recreate the stack on the worker selector when -you intentionally accept losing that monitoring data. A planned monitoring data -migration should be handled as a separate maintenance task with backup, -delete/recreate or storage migration steps, and post-restore checks. - -The older NodePort path is now reserved for special cases such as the local -registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` -support, but app traffic should normally enter through Traefik's MetalLB -address instead of per-app NodePorts. Before applying the edge stack, Jeannie -configures a Tailscale subnet router so the OCI edge can route to the Traefik -LoadBalancer address. The default route advertiser is the Debian control-plane -host because it can reach both Tailscale and the MetalLB address; set -`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. - -The cluster stack still supports worker-based subnet-route advertisement when -that is useful: - -```hcl -metallb = { - enabled = true - repository = "https://metallb.github.io/metallb" - version = "0.16.0" - namespace = "metallb-system" - address_pool = ["192.168.100.240-192.168.100.240"] - l2_advertisement_enabled = true - pool_name = "homelab-lan" -} - -traefik = { - enabled = true - repository = "https://helm.traefik.io/traefik" - chart_version = "40.2.0" - namespace = "traefik" - load_balancer_ip = "192.168.100.240" - ingress_class = "traefik" -} - -tailscale_subnet_routes = { - enabled = true - worker_key = "raspberrypi" - routes = ["192.168.100.0/24"] -} -``` - -DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for -the service hostnames land on the same edge host. For direct LAN testing, point -LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at -Traefik's address: - -```text -192.168.100.240 lab2025.duckdns.org -192.168.100.240 demos.lab2025.duckdns.org -192.168.100.240 n8n.lab2025.duckdns.org -192.168.100.240 prowlarr.lab2025.duckdns.org -192.168.100.240 sonarr.lab2025.duckdns.org -192.168.100.240 radarr.lab2025.duckdns.org -192.168.100.240 qbittorrent.lab2025.duckdns.org -192.168.100.240 kapowarr.lab2025.duckdns.org -192.168.100.240 suwayomi.lab2025.duckdns.org -192.168.100.240 maintainerr.lab2025.duckdns.org -192.168.100.240 grafana.lab2025.duckdns.org -192.168.100.240 argocd.lab2025.duckdns.org -``` - -Heimdall is external to Kubernetes. For direct LAN testing, use -`http://192.168.100.73:8082/`; public access still uses -`https://heimdall.lab2025.duckdns.org/` through the OCI edge. - -The edge stack uses Traefik as its backend by default and validates -`http://192.168.100.240:80/` before updating the edge containers. If Tailscale -subnet-route approval is not automatic in the tailnet policy, the edge deploy -will fail clearly instead of silently keeping a broken public path. - -For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of -remote SSH targets when more worker nodes exist. Set it to an empty string for a -single-node rebuild. - -## Adding Platform Tools - -Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. - -## Policy Guardrails - -`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security -policies in `Audit` mode. This gives the lab policy reports for unsafe workload -settings without blocking existing pods during the first rollout. After reports -are clean, individual policies can be promoted to `Enforce` in -`bootstrap/platform/main.tf`. - -`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for -the images built by this repo and pushed to the local registry. The policy -targets `192.168.100.73:30500/php-website:*` and -`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, -and audits whether the image has both: - -- a valid Cosign signature from the homelab signing key -- a signed SPDX SBOM attestation - -The private Cosign key, generated password, and downloaded Cosign binary live in -`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing -identity survives temporary checkout cleanup. Repo-local image state files still -live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses -the persistent Cosign key, publishes the public key into the Kyverno namespace as -`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs -the pushed images, attaches a signed SPDX SBOM predicate, and verifies both -artifacts before refreshing the workload Applications. The policy starts in -`Audit` mode so an existing live deployment can recover while the current images -are signed; after `./jeannie apps` verifies both images, change `validationActions` in -`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to -`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to -manage the key password externally; otherwise the Debian runner creates -`cosign/cosign.password` under the homelab state directory for non-interactive -runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override -those paths. - -## DNS Cache - -`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with -`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on -`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the -rollout compatible with the current kube-proxy iptables path without rewriting -kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service -CIDR or upstream DNS servers change. - -## MetalLB - -MetalLB is present in `bootstrap/platform` but disabled by default. Enable it -only after reserving a LAN IP range outside DHCP and outside any future OpenWrt -LAN pool: - -```bash -export TF_VAR_metallb='{ - enabled = true - repository = "https://metallb.github.io/metallb" - version = "0.16.0" - namespace = "metallb-system" - address_pool = ["192.168.100.240-192.168.100.250"] - l2_advertisement_enabled = true - pool_name = "homelab-lan" -}' -``` - -Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services -such as the website, demos, and n8n should be `ClusterIP` services behind -Ingress objects. The local registry remains a `NodePort` because the cluster -nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; -they run on the Debian Docker host. - -## Secrets - -Use SOPS with age for secrets that need to live in Git. The Debian host -bootstrap installs `age` and `sops`; then run: - -```bash -./jeannie secrets-init -./jeannie secrets-check -``` - -`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders -`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. -Commit `.sops.yaml`, but keep the private age key outside the repo. Operational -notes are in `docs/secrets.md`. - -## Edge Services - -The OCI jump box runs the public edge path: - -```text -nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP -``` - -Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. -Validate it before applying with: - -```bash -./jeannie tailnet-policy-check -``` - -The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and -deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in -`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` -file when the public host, SSH key, server name, backend Tailscale IP, or -Traefik backend address changes. - -The `/git/` route is intentionally different from the Kubernetes app routes: it -proxies to Gitea on the Debian host through Tailscale instead of Traefik. This -keeps public read-only source browsing available even when the cluster has been -destroyed. - -For the main website, nginx terminates TLS, serves cached HTML and static -assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS -sessions, and keeps upstream connections open to the Traefik backend. The -application side marks pre-rendered HTML as -`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS -under the stronger immutable asset policy. In local measurements, cached -single-resource requests are dominated by the remote TCP/TLS path, while a -reused HTTP/2 asset batch completes in roughly 60 ms after the connection is -established. - -Use the configured `server_name` in the browser, for example -`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser -certificate warning because the trusted certificate is issued for the hostname. - -The edge stack uses HTTP-01 validation and requests one certificate covering -`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains -under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must -be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive -expiry notices, or leave it empty to register without an email. Set -`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. - -Operational runbooks: - -- [Edge failures](docs/runbooks/edge-failures.md) -- [Gitea failures](docs/runbooks/gitea-failures.md) -- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) - -## Adding Apps - -Add Kubernetes manifests under `apps/` and register them in -`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and -self-healing for the app. - -The `heimdall` Argo CD app name is retained as a small route bundle for -Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can -prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes -workloads there. Use `./jeannie deploy-heimdall` for the dashboard -itself. - -The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is -exposed at `https://n8n.lab2025.duckdns.org`. It receives -`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing -Debian-host Ollama API for translation prewarming and batch jobs. - -The ARR/media stack is intentionally not deployed by the Kubernetes app -pipeline. It is better suited to a Docker Compose stack on the Debian host with -persistent volumes under `/data`, where large media paths, qBittorrent writes, -and service config directories can survive cluster rebuilds without OpenEBS -node-affinity or PVC migration concerns. The active host-level Compose -definition lives under `infra/arr-stack` and stores persistent data under -`/data/arr`. Historical Kubernetes manifests are archived under -`archive/apps/arr-stack` for migration reference only. - -## Storage - -OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use -retained local PV paths such as `/data/openebs/local/registry`; these paths are -intentionally outside kubeadm reset paths so data can survive cluster -destroy/create cycles. Those critical volumes are declared explicitly as -retained local PVs so a rebuilt cluster binds back to the same host paths -instead of creating fresh directories. - -For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the -standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath -data still lives under `/data/openebs/local`, and larger service data such as -Gitea, Ollama models, and app volumes should stay under `/data`. - -## Gitea - -Gitea is external bootstrap infrastructure. It runs on the Debian host as an -always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a -Kubernetes workload. This keeps Git available when the Kubernetes cluster is -destroyed and rebuilt. - -The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. -Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. - -Public source browsing stays available through -`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users -can view public repositories, so the blog can link to code read-only while -writes still require an authenticated Gitea account. - -## Heimdall - -Heimdall is external bootstrap infrastructure. It runs on the Debian host as an -always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not -as a Kubernetes workload. This keeps the lab navigation page available when the -cluster is destroyed, rebuilding, or unhealthy. - -Deploy or refresh it from the Debian server with: - -```bash -./jeannie deploy-heimdall -``` - -`./jeannie up` also runs this step during the independent host -services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path -is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. -Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar -container that upserts repo-managed tiles without deleting manually added ones. - -The Debian bare repo remains the GitOps mirror: - -```text -/home/jv/git-server/my-homelab-configs.git -``` - -Argo CD consumes that Debian mirror through the default `gitops_repo_url`. -Gitea Actions pushes the `main` commit into the mirror before running the -selected deploy command. - -The platform bootstrap registers the Argo CD repository secret and the SSH host -key for the Debian GitOps mirror. If Argo CD reports -`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host -key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, -restart `argocd-repo-server`, and hard-refresh the affected Application. - -Deploy or refresh the external Gitea container from the Debian host with: - -```bash -./jeannie deploy-gitea -``` - -## Gitea Backups - -`./jeannie up` installs a Debian-host systemd timer named -`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured -Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the -dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default -retention is 30 days. - -The same install step also creates `homelab-gitea-restore-drill.timer`. The -monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it -to a temporary directory, records a report under -`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It -does not write into the live Gitea data directory. - -Run a manual backup from the Debian server with: - -```bash -./jeannie backup-gitea -``` - -Run the restore drill manually with: - -```bash -./jeannie drill-gitea-restore -./jeannie drill-restore -./jeannie drill-pihole-restore -``` - -`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed -config inputs, the latest OpenTofu state backup archive when present, and the -GitOps rebuild path. It does not replace running services or mutate Kubernetes. - -Useful checks: - -```bash -./jeannie backup-status -systemctl list-timers homelab-gitea-backup.timer -systemctl list-timers homelab-gitea-restore-drill.timer -sudo systemctl start homelab-gitea-backup.service -ls -lh /home/jv/backups/gitea -ls -lh /home/jv/backups/gitea-restore-drills -``` - -## Gitea Actions - -This repo includes a Gitea Actions workflow at -`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and -`main`, and deploys only from `main`. That keeps the promotion path simple: - -```text -dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync -``` - -The workflow targets a repository-scoped Debian host runner with the label -`homelab-debian`. - -Run the same validation locally before promoting: - -```bash -./jeannie validate -``` - -## Identity and access audit - -Run a read-only access inventory from the Debian server with: - -```bash -./jeannie access-audit -``` - -The audit reports configured SSH targets and key file modes, Git/Gitea remotes, -Tailscale ACL policy validation, current Kubernetes authorization, broad -cluster-admin bindings, automation service accounts, and Gitea runner status. -It does not create users, tokens, kubeconfigs, or keys. Use it to move toward -separate identities: - -- admin kubeconfig only on the Debian control host -- read-only kubeconfig for dashboards and audits -- separate service accounts for automation -- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge -- repo-managed Tailscale ACLs - -The read-only Kubernetes identity is managed in `apps/access-control`. After -that app syncs, generate a separate read-only kubeconfig on the Debian host: - -```bash -./jeannie kubeconfig-readonly -``` - -The workflow only blocks automatic deploy for external Gitea service -changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, -`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` -functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by -default: Actions runs `./jeannie doctor-versions` when the Debian runner already -has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; -if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no -kubeconfig exists it also runs `./jeannie rebuild-cluster`. - -Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` -on the runner to force one path. - -`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key -with the Gitea repository and switches the Debian working copy's `gitea` remote -to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key -is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a -different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH -access unchanged. The Actions deploy job uses the checked-out Actions workspace -as the source commit, updates the first available persistent checkout from -`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or -`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the -Actions workspace. It does not need SSH read access back to Gitea. - -Enable Actions for the repository in Gitea, then create a repository-level runner -token from: - -```text -https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners -``` - -Register and start the Debian runner from the Debian server: - -```bash -cd ~/my-homelab-configs -GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner -``` - -The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and -uses a host label instead of a Docker job container because deployment needs the -Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. - -The deployment job is non-interactive. User `jv` must be able to run `sudo -n -true` on the Debian host for deployment commands that require sudo. - -Useful checks: - -```bash -systemctl status homelab-gitea-runner.service -journalctl -u homelab-gitea-runner.service -n 100 --no-pager -``` - -## Renovate - -`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu -providers, Helm chart versions, and the pinned tools used by the Gitea Actions -workflow. Renovate should open reviewable update branches or PRs only; it must -not auto-merge infrastructure changes. Keep app-only dependency updates on the -normal Gitea Actions path, and run `./jeannie up` manually on the Debian server -for platform or provisioning updates. - -## Destructive Rebuilds - -`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico -links, iptables rules, and local OpenTofu state. It does not delete retained data -under `/data/openebs/local`. - -For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH -targets. It defaults to an empty string so worker nodes are not cleaned unless -you explicitly include them. - -## Website App - -The website is a PHP app under `apps/website`. It includes a home page, CV page, -blog page, and demos page, plus a lightweight translation flow backed by Redis, -n8n, and Ollama. -Static language files live in `apps/website/lang`; `en.php` and `nah.php` are -curated source files, with the Nahuatl home page intentionally biased toward as -many Nahuatl words as possible while keeping technical terms understandable. -Unsupported browser languages use the same-origin `/translate.php` endpoint, -which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the -browser never calls the private Ollama IP directly. Redis caches per-string -translation results in the `website-production` namespace, while n8n is -available for translation prewarming and batch workflow jobs. The default model -is the custom `website-translator` Ollama model defined in -`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved -through `save_lang.php` on the website PVC, and `translate.php` emits structured -logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. - -Create or refresh the Ollama model on the Debian server before deploying a -website image that points at it: - -```bash -./jeannie website-translation-model -``` - -Ollama must also listen on the Debian host LAN address so Kubernetes pods on -other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: - -```bash -./jeannie website-ollama-listen -``` - -The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` -is true: it installs the service, stores models under `/data/ollama/models`, -binds the API to the configured LAN listener, and pulls the lightweight -`ai_gateway.model`. On an existing host, apply just this setup with: - -```bash -./jeannie ollama-setup -``` - -`jeannie` also has a backstage local AI helper for doctor commands. -It is not a separate CLI action; when `ai_gateway.enabled` is true in -`homelab.yml`, doctor commands try the configured Ollama model and silently -skip the helper if Ollama is down. The default lightweight model is: - -```bash -ollama pull qwen2.5:0.5b -``` - -Disable it for low-CPU sessions with: - -```bash -LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge -``` - -Build the local homelab knowledge index separately from the main deployment: - -```bash -./jeannie ai-index -./jeannie ai-check -./jeannie ask "how do I check edge to Gitea?" -``` - -The index is built from the non-secret source list in -`infra/ai/knowledge-sources.txt`, skips paths in -`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. -Doctor commands use it as extra context when it exists, but infrastructure -deployment does not depend on it. -Use `ask` as the runbook and command finder. It returns the closest indexed -docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after -retrieval. - -The CV page has two client-side presentation modes: - -- `Elegant`: dark, minimal, terminal-inspired styling with a square profile - image and light green console text. -- `Fancy`: centered circular profile image, cursive orbit text, and a - cursor-following portrait rotation effect. - -The Demos page is a catalog in the PHP website. The actual demo applications are -served from a separate `demos-static` artifact under `apps/demos-static` and are -published through the `demos-static` Argo CD application. Public traffic reaches -them through the edge path at `/demo-apps/`. - -`./jeannie up` builds and pushes two independent images: - -- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD - as a Kustomize image override -- `demos-static:latest` from `apps/demos-static` -- Cosign signatures and signed SPDX SBOM attestations for both pushed images - -The website manifest keeps the stable base image name `php-website:bootstrap`. -During bootstrap, `jeannie` hashes `apps/website`, builds -`/php-website:src-`, exports that exact reference through -`TF_VAR_website_image_ref`, and the Argo CD Application applies it through -Kustomize. This keeps the GitOps source generic while the deployed image remains -immutable. The Kyverno supply-chain policy mutates admitted pods to the verified -image digest, so the workload runs the same digest that was signed and attested. - -After `./jeannie apps`, the live deployment image should be a content-hash tag, -for example `192.168.100.73:30500/php-website:src-...`. If it still shows -`php-website:latest`, Argo CD has not rendered the current Application source. -Check the `website-production` Application source, sync status, and repository -access before restarting pods. - -The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs -private, browser-only image compression and conversion using native Canvas APIs. -Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled -to WebAssembly with a TypeScript UI so the codec work stays fast and still -avoids backend uploads. - -The demos are designed to be local-first so the current cluster can serve them -from any Linux app node without turning either pod into an application server. -The website pod serves the portfolio shell and the `demos-static` pod serves -static demo bundles; CPU-heavy work runs in the visitor's browser. Because the -deployments can run on either Debian or ARM workers, avoid bundling large ML -models, server-side WebSocket probes, or backend video transcoders into either -image. If those demos become production-grade, lazy load model assets in the -browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 -Plus. - -Current demo inventory: - -- Client-side media cruncher: image conversion/compression with Canvas; future - Rust/Wasm codec path for video. -- Internet quality visualizer: live Canvas graph for latency, jitter, and - stability using same-origin browser probes; a dedicated WebSocket echo endpoint - would be the production version. -- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and - local text-log filtering. -- Architecture simulator: click-driven load, crash, and auto-scale simulation. -- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB - conversions. -- Privacy-first redactor: local image redaction prototype; future - onnxruntime-web plus quantized YOLO or face model path. -- Local sentiment sandbox: lightweight local sentiment, keyword, and summary - prototype; future Transformers.js/ONNX path. -- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, - and retraining. - -The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. -Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by -the app image build and with the image globs in -`apps/supply-chain-policy/local-registry-image-policy.yaml`. - -Keep the `.terraform.lock.hcl` files committed. They pin provider selections and -make bootstrap behavior reproducible across nodes and rebuilds. +Run `./jeannie up` and `./jeannie nuke` only from the Debian homelab server. diff --git a/README.md.tmpl b/README.md.tmpl index c1089a5..254077f 100644 --- a/README.md.tmpl +++ b/README.md.tmpl @@ -2,324 +2,9 @@ -This repo bootstraps a hybrid kubeadm cluster and then hands app delivery to -Argo CD. +This repo bootstraps a hybrid kubeadm cluster and then hands app delivery to Argo CD. -## Architecture - -The lab is intentionally small but production-shaped: - -- a Debian amd64 host runs the kubeadm control plane and local deployment tools -- a Raspberry Pi arm64 node runs selected workloads -- the Debian host also runs the always-on Gitea Docker service outside - Kubernetes -- the Debian host keeps a bare GitOps mirror under - `/home/jv/git-server/my-homelab-configs.git` -- a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker - templates -- OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge -- Argo CD continuously reconciles Kubernetes manifests from this repo -- a local registry stores the website and demos images built for the worker - architecture -- the website image keeps Apache as the public server, uses PHP-FPM with - OPcache for dynamic endpoints, and pre-renders read-heavy pages to static - HTML during the image build -- SOPS with age is the committed secret-management path for future encrypted - Kubernetes secrets -- local Jeannie/RAG eval cases exercise troubleshooting explanations and - retrieval grounding without touching live infrastructure -- an OCI jump box provides the public edge path back into the homelab over - Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream - keepalive - -## Inventory - -The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, -Tailscale IPs, public hostnames, ports, storage paths, and service ownership -there before copying values into scripts, OpenTofu variables, runbooks, or -service docs. Secrets and tokens do not belong in that file. - -`./{{ main_script }}` reads `homelab.yml` at startup and exports matching OpenTofu -variables for the edge, provisioning, cluster, platform, and apps stacks. This -keeps values such as the Debian LAN IP, Debian Tailscale IP, RPi LAN IP, OCI IP, -Traefik MetalLB IP, and domain name from drifting across scripts and templates. -`scripts/render-docs` also renders [docs/service-catalog.md](docs/service-catalog.md) -and the static service catalog page at -`apps/demos-static/public/homelab-catalog/index.html` from the same inventory. - -Run `./{{ main_script }} up` and `./{{ main_script }} nuke` only from the Debian homelab server. The -script intentionally refuses to run from non-Debian machines so a laptop cannot -accidentally modify the cluster. - -## Flow - -1. `bootstrap/provisioning` - - prepares a Debian server as a PXE and preseed service for arm64 VMs - - serves Debian 13 arm64 netboot assets through TFTP and HTTP - - exposes a PXE rescue menu and HTTP rescue page for manual repair paths - - creates a golden image install path with Kubernetes, containerd, - qemu-guest-agent, cloud-init, and storage client packages ready - - is driven by `./{{ main_script }} up` when Pimox is reachable, without changing - Orange Pi host networking - -2. `bootstrap/cluster` - - creates the kubeadm control plane on the Debian amd64 node - - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes - - configures Calico-compatible pod CIDR - - configures containerd to pull from the in-cluster NodePort registry - - creates retained host directories under `/data/openebs/local` - -3. `bootstrap/platform` - - installs a minimal Calico deployment through the Tigera operator - - installs NodeLocal DNSCache for node-local DNS query caching - - installs MetalLB for LAN `LoadBalancer` services - - installs Traefik as the single Kubernetes ingress entry point - - installs OpenEBS - - creates `openebs-hostpath-retain` - - installs Argo CD - - installs Kyverno with audit-first baseline Pod Security policies and - homelab image signature/SBOM admission controls - - registers the private GitOps repo without storing the SSH private key in - Terraform state - -4. `bootstrap/apps` - - registers Argo CD Applications from the `applications` map - - passes the website image produced by the build step into Argo CD as a - Kustomize image override - - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM - listens on loopback inside the container, OPcache is enabled, and - pre-rendered HTML handles the read-heavy pages - - default apps are `container-registry`, `website-production`, - `demos-static`, the `heimdall` route bundle, `n8n`, and - `supply-chain-policy` - -5. `bootstrap/edge` - - connects to the OCI jump box - - uploads nginx, HAProxy, Varnish, and Squid configs - - obtains and renews Let's Encrypt certificates for the configured hostname - - runs the edge cache/proxy chain with Docker Compose - - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream - keepalive, and cache headers for the public website path - -## Prerequisites - -On the Debian host: - -- OpenTofu -- Docker with Buildx -- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes - minor, currently `v1.36` -- SSH access to worker nodes -- SSH access to the OCI edge host -- Go toolchain for Jeannie core helpers; `./{{ main_script }} up` installs - `golang-go` on the Debian host if it is missing -- enough persistent storage for `/data/openebs/local` and Docker's data root - -After a clean Debian reinstall, bootstrap the host package set and desktop -configuration first: - -```bash -sudo apt-get update -sudo apt-get install -y git ansible sudo -git clone /home/jv/my-homelab-configs -cd /home/jv/my-homelab-configs -ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml -``` - -If you kept the private app-config archive generated from the old Debian host, -pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. -Details are in `bootstrap/host/README.md`. - -The default kubeconfig path is `/home/jv/.kube/config`. Override it with -`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. - -## Implementation Language Policy - -Jeannie keeps `./{{ main_script }}` as the stable command entrypoint. New durable -read-only orchestration, report rendering, inventory parsing, status/doctor -logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because -it ships as one small binary and is easier to test than Bash. Bash remains for -thin compatibility wrappers and remote shell glue. Python remains acceptable for -AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the -clear fit. New repo features should choose the smallest language that fits the -task instead of defaulting to Bash. - -## Version Discipline - -The lab should converge on one Kubernetes minor across the API server and every -kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers -match the Debian control plane and Raspberry Pi worker instead of staying on an -older template. Avoid treating a three-minor kubelet skew as normal steady -state: it is only a temporary compatibility window and blocks the next control -plane upgrade. - -Check node versions from the Debian host: - -```bash -./{{ main_script }} doctor-versions -``` - -The check prints the API server, kubelet versions, kubelet minors, and containerd -runtimes. It exits nonzero when kubelet minors differ from the API server minor -or when containerd major versions are mixed. The Gitea Actions auto path runs -this check before app deploys; kubelet minor drift switches the job to -`./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced -from the pinned image. For Pimox workers, prefer rebuilding from the corrected -golden template and rejoining the worker over ad hoc live package upgrades. - -## AI Evals - -The repo includes a local eval harness for Jeannie explanations and homelab RAG -grounding. It uses fixtures and allowlisted read-only commands, so it is safe to -run before changing the lab: - -```bash -./{{ main_script }} ai-evals run -``` - -Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases -under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior -should become repeatable. - -## Prompt Injection Lab - -The defensive security training path includes local prompt-injection fixtures -for hostile runbooks, tool output, dashboard annotations, and command arguments. -Run it without touching live infrastructure: - -```bash -./{{ main_script }} prompt-injection-lab run -``` - -Cases live under `security/prompt-injection-lab` and are meant to become -regression tests for future Jeannie AI features. - -## Incident Commander - -The repo also includes a read-only incident commander trainer. It classifies -pasted failure text, prints a likely hypothesis, links the right runbook, and -lists the next read-only Jeannie commands: - -```bash -./{{ main_script }} incident replay -./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" -``` - -Incident fixtures live under `infra/incident-commander`. Use them to turn -repeat outages into repeatable triage behavior before adding any self-healing. - -## Safety Cases - -Before high-blast-radius changes, generate a safety case from the Git diff: - -```bash -./{{ main_script }} safety-case --since HEAD~1 -``` - -It summarizes risk, blast radius, required validation, rollback, and forbidden -actions. The implementation lives under `infra/safety-cases` and reuses the -same impact rules as `./{{ main_script }} impact`. - -## Agent Sandbox - -Future Jeannie AI actions should be checked before execution. The local sandbox -policy starts with a read-only command classifier: - -```bash -./{{ main_script }} agent-sandbox check "./{{ main_script }} status" -./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" -``` - -Policy is stored in `infra/agent-sandbox/policy.tsv`. - -## AI Workload Scheduling - -For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox -workers, and the RPi based on CPU, memory, architecture, tailnet access, and -control-plane policy: - -```bash -./{{ main_script }} ai-scheduler recommend small-rag -./{{ main_script }} ai-scheduler simulate -``` - -The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a -real Kubernetes controller. - -## Memory With Provenance - -Jeannie can return cited retrieval context and check whether the local RAG index -is stale: - -```bash -./{{ main_script }} ask --citations "how do I recover the cluster?" -./{{ main_script }} ai-memory check -``` - -This keeps local AI answers grounded in repo files and runbooks. - -## Red/Blue Loop - -Safe red-team/blue-team checks are grouped under one command: - -```bash -./{{ main_script }} red-blue plan -./{{ main_script }} red-blue run --local -``` - -Local mode runs deterministic prompt-injection and eval checks. Live scans are -listed but gated so they can run from the Debian host when intended. - -## Canary Promotion - -The promotion helper defines release gates before production changes: - -```bash -./{{ main_script }} promote plan -./{{ main_script }} promote validate -./{{ main_script }} promote rollback website-production -``` - -The current version is intentionally conservative: it validates local gates and -prints rollback steps while dev namespace automation remains explicit future -work. - -## Guarded Self-Healing - -Jeannie can turn status findings into an experimental self-heal plan: - -```bash -./{{ main_script }} status --heal-plan -./{{ main_script }} status --heal -./{{ main_script }} heal apply --yes -``` - -Normal `status` remains read-only. The heal planner maps known findings to -guarded actions, chooses one highest-impact auto-eligible finding, checks the -proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that -one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie -can reassess before touching the next issue. If a specific Pimox worker is -NotReady, heal targets that worker with `./{{ main_script }} workers restart N` -instead of repeating a broad cluster start. After an action runs, Jeannie waits -for the target condition to clear and reports `healed` or `still failing`. -Failed heal actions are placed on a short cooldown so Jeannie does not loop on -the same VM restart; the next plan escalates to diagnostics instead. -Destructive actions remain blocked. - -## Model Behavior Observatory - -Stable prompts track local model behavior, latency, refusal behavior, and -tool-correctness signals over time: - -```bash -./{{ main_script }} model-observe prompts -./{{ main_script }} model-observe run --offline -./{{ main_script }} model-observe report -``` - -Use `--live` only when Ollama is available. - -## Deploying +## Quick Start From the Debian server: @@ -328,1162 +13,21 @@ cd ~/my-homelab-configs ./{{ main_script }} up ``` -The script first deploys external Gitea to the Debian host with Docker Compose -under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays -outside the Kubernetes rebuild blast radius. It then detects the -Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are -available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 -arm64 template, creates or reuses one worker VM clone, discovers the guest IP -through qemu-guest-agent, and passes that worker into the cluster layer. It then -applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the -local registry, builds the website and demos images when their source changed, -pushes them to the registry, recreates pods only after a new image is built, and -applies the edge stack. +## Documentation Hub -Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set -`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline -keeps the template on its configured `local` storage, creates new worker VM -clones on `opi5_ssd` by default when that Pimox storage is available, -checks that the Pimox bridge already exists, refuses `local` as worker clone -storage, and refuses to edit Orange Pi host networking. Keep durable data on -the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks -as rebuildable worker capacity. +The detailed technical specifications have been moved to the `/docs` directory: -`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns -worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set -`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker -must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID -`9010` and VMID `9011`. +- [Architecture & Infrastructure](docs/architecture.md) - Cluster layout, Networking, and Edge path. +- [Deployment Guide](docs/deployment.md) - Prerequisites, bootstrap flow, and validation. +- [AI Operations](docs/ai-ops.md) - Jeannie AI tools, Evals, and Safety cases. +- [Application Ops](docs/app-ops.md) - App delivery, Gitea, and Website management. +- [Service Catalog](docs/service-catalog.md) - Live inventory of all hosts and services. -Pimox workers can join the tailnet during cluster worker bootstrap. Keep the -ephemeral or pre-authorized auth key outside Git, either in -`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in -`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: +## Inventory -```bash -LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up -``` +The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, Tailscale IPs, public hostnames, ports, storage paths, and service ownership there. -For existing workers, prefer the Jeannie wrapper because it reads the generated -worker keys and exports the correct OpenTofu variables: +`./{{ main_script }}` reads `homelab.yml` at startup and exports matching OpenTofu variables. +`scripts/render-docs` renders the documentation and the static service catalog page. -```bash -./{{ main_script }} workers tailnet -``` - -When applying `bootstrap/cluster` directly, use the OpenTofu variable name -instead of the Jeannie `LAB_` wrapper: - -```bash -TF_VAR_worker_tailscale_enabled=true \ -tofu -chdir=bootstrap/cluster apply \ - -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" -``` - -Worker Tailscale defaults to `--accept-routes=false` because these VMs already -live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can -steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT -from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox -workers can reach tailnet-only services. Set -`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume -tailnet-advertised routes that do not overlap the local LAN. Set -`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through -`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox -worker tailnet egress against -`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and -`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. - -OpenWrt firewall VM automation is available as a standalone command because it -attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` -already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM -SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, -imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores -the VM disk on the selected non-local Pimox storage. It leaves the VM stopped -and not enabled for host boot by default. It does not use the Debian Kubernetes -golden-node template for OpenWrt. - -The website and demos images default to `linux/amd64,linux/arm64` because both -deployments can schedule on either the Debian host or ARM workers. Override with -`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally -restricting node placement. - -Build metadata is written under `.lab/` so repeat runs can skip the website -or demos image build when the source hash, platform, image reference, and -registry manifest still match. - -The website build is intentionally performed on the Debian homelab path and the -Gitea Actions runner, not on a laptop. The current image uses -`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables -OPcache for immutable image deploys, and runs -`apps/website/render-static-pages.sh` during the image build. That render step -produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and -`/homelab-tree.php` in the static languages. Runtime PHP remains for write and -translation endpoints such as `save_idea.php`, `visitor_ideas.php`, -`translate.php`, and `save_lang.php`. - -Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the -service is already managed manually. The default Gitea target is -`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, -and SSH port `32222`. - -Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` -runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker -root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, -Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory -or host storage/networking: - -```bash -./{{ main_script }} preflight -``` - -If the Debian Docker root check fails after a reinstall, repair it before -running the full pipeline: - -```bash -./{{ main_script }} fix-debian-docker-root -``` - -When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but -the Kubernetes API is down, it starts the saved runtime before applying cluster -changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally -want to destroy and recreate the cluster. - -## Validation - -Useful checks after a rebuild: - -```bash -export KUBECONFIG=/home/jv/.kube/config - -kubectl get nodes -kubectl top nodes -kubectl -n argocd get applications -kubectl -n container-registry get pods -kubectl -n website-production get pods -o wide -kubectl -n demos-static get pods -o wide - -ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' -ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' - -docker info --format '{{.DockerRootDir}}' -df -h / /data /data/openebs/local /data/docker -``` - -The website should be reached through the configured public hostname, not the raw -OCI IP address, because the Let's Encrypt certificate is issued for the -hostname. - -Run a read-only health snapshot from the Debian server with: - -```bash -./{{ main_script }} help -./{{ main_script }} status -./{{ main_script }} capacity -./{{ main_script }} capacity-advisor -./{{ main_script }} capacity-limits -./{{ main_script }} recover-plan -./{{ main_script }} recover-power --dry-run -./{{ main_script }} impact --since HEAD~1 -./{{ main_script }} review-last-change -./{{ main_script }} scorecard -./{{ main_script }} explain scorecard -./{{ main_script }} gitops-status -./{{ main_script }} cert-check -./{{ main_script }} grafana-dashboards list -./{{ main_script }} release-snapshot -./{{ main_script }} backup-status -./{{ main_script }} synthetic-checks -./{{ main_script }} resource-budget -./{{ main_script }} control-plane status -./{{ main_script }} artifact-cache status -./{{ main_script }} blockchain-devnet status -./{{ main_script }} blockchain-test -./{{ main_script }} blockchain-wallet instructions -./{{ main_script }} golden-ledger check -./{{ main_script }} route-inventory -./{{ main_script }} workers list -./{{ main_script }} change-journal list -./{{ main_script }} map -``` - -`help` prints the grouped command reference in pipeline order, from inventory -and bootstrap through cluster lifecycle, reports, observability, recovery, -access, apps, security, AI, and destructive commands. - -`status` uses the reusable report UI and defaults to showing only failures and -warnings so the terminal stays readable. Use `./{{ main_script }} status --all` -to show every check, `--details` for row details, `--json` for structured -output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ -Tailscale/HTTP dump. The cascade includes "what broke" signals such as -deployment readiness, pod restarts, node pressure, disk pressure, Traefik -5xx/404 log evidence, and recent Gitea errors. - -High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node -readiness, pod restart storms, unavailable deployments, node/PVC storage -pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape -coverage. Keep this set small so alerts remain actionable. - -Application egress is controlled with Kubernetes NetworkPolicies where the lab -owns the workload. The website namespace defaults to denied egress and allows -only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is -isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS -for controlled practice targets. - -`capacity` is the placement report for the hardware you already have. By -default it uses the compact report renderer and shows summarized Debian, -Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use -`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or -`--verbose` for the older full capacity dump. - -`capacity-advisor` interprets capacity, resource-budget, and control-plane -placement signals to recommend whether to add Pimox VMs, taint Debian, or fix -resource definitions first. - -`capacity-limits` prints suggested Kubernetes `resources.requests` and -`resources.limits` YAML snippets for containers missing them. It uses -`infra/resource-recommendations.yml` as a conservative starting profile and does -not edit manifests. - -`recover-plan` prints the disaster recovery order and lightweight prerequisite -checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and -edge verification. - -`recover-power` runs the post-outage recovery sequence in dependency order: -Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and -edge/public checks. Use `recover-power --dry-run` to preview the sequence. - -`impact` explains what a change may touch before you apply it. Use -`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths -like `bootstrap/edge` and `apps/website`. - -`review-last-change` summarizes the last commit, runs impact analysis for the -touched files, and prints validation plus rollback hints. - -`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster -health, edge health, capacity pressure, security posture, public certificates, -inventory, and docs freshness into a compact pass/warn/fail report. - -`explain` is the read-only follow-up for warnings, failures, and corrective -output. It can run safe report commands itself, such as -`./{{ main_script }} explain scorecard`, `status`, `capacity`, or -`resource-budget`, and it can explain captured output with -`./{{ main_script }} explain output < saved-output.txt`. It lists what each -finding means, diagnostic commands, whether a Jeannie fix command already -exists, and risk/auto-fix safety. - -Report-oriented commands share the reusable UI module in `scripts/report-ui` -and renderer in `scripts/report-render`. Commands emit status TSV rows, and the -renderer prints compact grouped summaries by default, with details or JSON -available for future TUI/web output. Rows may include separate `fix`, `graph`, -and `query` fields so a warning can point to both the next `jeannie` command and -the matching Grafana panel or Explore query. - -`gitops-status` prints a focused Argo CD view: application sync and health, -out-of-sync or degraded apps, repository secret presence, recent Argo CD events, -and recent repo-server/application-controller errors. - -`grafana-dashboards` manages repo-provisioned Grafana dashboards under -`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions -them through the platform stack; `./{{ main_script }} grafana-dashboards apply` -rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional -blockchain devnet dashboard uses metrics from -`infra/blockchain-devnet/metrics/exporter.py`. - -`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea -HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. - -`release-snapshot` writes a timestamped pre-change report under the homelab -state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, -public URL status, and pointers to the latest Gitea/OpenTofu backups. - -`backup-status` reports freshness for Gitea backups, OpenTofu state backups, -restore drill reports, and repo-managed Pi-hole restore inputs. The main -`status` cascade includes this as a warning signal. - -`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, -Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The -registry push/pull smoke test is opt-in with -`LAB_SYNTHETIC_REGISTRY_PUSH=true`. - -`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` -against Kubernetes pod requests/limits and Debian host disk/memory signals. Use -it before adding heavier workloads so the HP laptop remains usable. By default -it summarizes missing requests/limits by namespace; use -`./{{ main_script }} resource-budget --details` for every pod/container. - -`control-plane` shows or manages the Debian Kubernetes control-plane scheduling -taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` -to keep normal app/data-plane pods off the Debian host while control-plane and -DaemonSet workloads continue using their normal tolerations. - -`artifact-cache` manages an optional Debian-host cache stack in -`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through -cache. Use `artifact-cache instructions` for client configuration snippets. - -`blockchain-devnet` manages an optional local Ethereum Anvil devnet under -`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability -learning only; never use real seed phrases or mainnet keys. - -`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts -with a simple `Counter` contract and ownership tests so there is a known-good -baseline before practicing vulnerable contracts. - -`blockchain-wallet` provides disposable dev-wallet generation, address -derivation, and message-signing practice. It is documented in -`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. - -`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the -reviewable record of template VMID, storage, OS release, Kubernetes pins, -runtime versions, and build metadata for Pimox worker golden images. - -`route-inventory` reports Kubernetes ingress hosts, paths, backend services, -TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and -missing monitors are easier to spot. - -`workers` provides named Kubernetes/Pimox worker lifecycle operations: -`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and -`rebalance`. -Use `recreate-plan ` before replacing one broken Pimox worker so the -cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. - -`change-journal` lists local journal entries written before risky commands such -as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, -`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live -under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, -dirty file count, and command details. - -`map` prints a dependency map from the canonical inventory, covering public -entry, GitOps automation, cluster foundation, DNS, observability, and security -learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. - -Focused doctor commands run narrower read-only checks. They use the same -problems-first UI as `status`; add `--all`, `--details`, `--json`, or -`--verbose` when you need more context: - -```bash -./{{ main_script }} doctor-edge -./{{ main_script }} doctor-gitea -./{{ main_script }} doctor-rpi -./{{ main_script }} doctor-cluster -``` - -`doctor-cluster` also checks whether Pimox workers have Tailscale installed, -`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet -probe service. - -## RPi Services - -The Raspberry Pi runs lightweight always-on services outside Kubernetes from -`infra/rpi-services`: - -- Pi-hole on DNS port `53` and web port `8081` -- Unbound as Pi-hole's first upstream resolver -- Uptime Kuma on port `3001` - -Deploy or refresh them from the Debian server with: - -```bash -./{{ main_script }} rpi-services -``` - -`./{{ main_script }} up` also runs this step unless -`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is -expected under `/nvme-storage/docker` when that path is an active writable mount point. If -the NVMe is missing or resets, it rewrites Docker's `data-root` to -`/var/lib/docker` and restarts Docker so DNS can still come up from the -SD/root filesystem. - -Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can -continue resolving if the Unbound container stops. Repo-managed Pi-hole state -lives in `infra/rpi-services/adlists.txt`, -`infra/rpi-services/local-dns-records.txt`, -`infra/rpi-services/cname-records.txt`, and -`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without -deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into -a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` -is not provided, the bootstrap generates one and keeps it in -`/opt/homelab-rpi-services/.env`. -Uptime Kuma monitor seeds live in -`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first -Kuma user exists. -`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, -direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and -whether Docker is currently using `/nvme-storage/docker` or has fallen back to -`/var/lib/docker`. - -## Adding Nodes - -For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 -template and worker VM clones automatically when the Orange Pi storage path is -healthy. Defaults are intentionally tied to the observed host: Pimox SSH host -`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two -8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, -and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye -on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that -host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` -entries so native `qm` CPU topology is the only `-smp` argument. Set -`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe -that was previously attached to the Orange Pi now belongs to the HP Debian -laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox -datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and -override variables are in -`bootstrap/provisioning/README.md`. - -Worker indexes are stable. Index `1` maps to VMID `9010`, node name -`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and -so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while -allowing higher indexes to be automated. - -Worker networking defaults to static because the ISP modem DHCP server does not -answer Pimox worker VM DHCP requests behind `vmbr0`. Set -`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM -traffic. Static mode uses -`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, -`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and -`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. -For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions -for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` -otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before -adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` -only when the Pimox host firewall already permits bridged worker IP traffic. -Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under -`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. -Worker NIC MAC addresses default to the VMID-derived deterministic format used -by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` -only when you explicitly want Pimox to generate the MAC. -Worker creation fails faster than template creation: static guest-network -configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, -worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, -and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. -Static guest-network validation skips the extra LAN ICMP probe by default -because the ISP modem may not answer worker VM pings. Set -`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host -should be required during qemu-guest-agent network setup. After -qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by -default and lets SSH be the reachability check. Set -`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. -`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. -Before `./{{ main_script }} apps` changes any application resources, Jeannie -requires every generated Pimox worker to be `Ready` and expose at least two -Kubernetes CPUs. Override the threshold with -`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set -`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately -non-Pimox deployment. - -Long-running `./{{ main_script }} up` steps keep compact output by default, but -the active step line refreshes with elapsed time and the latest log line. Set -`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or -`JEANNIE_VERBOSE=true` to stream full command output. -Independent host services run through the Go DAG runner during `up`: Gitea -deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks -can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential -execution while troubleshooting, or raise it above the default `4` if the lab -can absorb more concurrent work. - -Run a full cluster rebuild from the Debian server with: - -```bash -./{{ main_script }} rebuild-cluster -``` - -That path preserves external Gitea, rebuilds the Pimox template -with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and -8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM -does not support `qm --affinity`; Jeannie instead starts the generated KVM -command under CPU set `4-7` and verifies every worker reports two guest CPUs -before Kubernetes workloads proceed. -The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` -does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. - -To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, -PV data, or VM disks, run: - -```bash -./{{ main_script }} stop-cluster -``` - -This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when -available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully -shuts down automated Pimox worker VMs. It intentionally leaves the host -`containerd`/Docker stack running so external Docker Compose services such as -Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove -state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want -to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` -to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a -hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, -`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to -select VM workers. Resume the runtime with: - -```bash -./{{ main_script }} start-cluster -``` - -Kubernetes worker generation defaults to automated Pimox workers only. The -Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` -is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless -`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, -add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: - -```hcl -worker_nodes = { - raspberrypi = { - host = "192.168.100.89" - user = "jv" - node_name = "raspberry" - ssh_key_path = "/home/jv/.ssh/id_ed25519" - } -} -``` - -Stateful apps currently pin retained local PVs to the `debian` node. Move or -duplicate those PV manifests when you want storage on another node. - -## Workload Placement - -`bootstrap/cluster` labels nodes with homelab placement metadata: - -- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` - shows `worker` instead of `` in the ROLES column -- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and - `homelab.dev/workload-class=control-plane` on the Debian control plane -- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and - `homelab.dev/workload-class=edge` on the Raspberry Pi worker -- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and - `homelab.dev/workload-class=platform` on automated Pimox worker clones when - those workers are enabled - -Override `control_plane_node_labels`, `worker_node_labels`, -`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when -the physical layout changes. The current website, demos, and registry manifests -are not moved automatically because the public NodePort path and retained -OpenEBS hostpath PVs are node-local. Move workloads only after their storage and -edge path are ready on the target node. Gitea is outside Kubernetes and is moved -by changing the Raspberry Pi Docker install target instead. - -The stateless platform controllers are pinned to Pimox worker nodes through -`homelab.dev/workload-class=platform` and include hostname topology spread plus -preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and -kube-state-metrics scheduling does not collapse onto the first worker that joins. -PVC-backed monitoring StatefulSets are intentionally treated separately because -their retained OpenEBS hostpath volumes are node-local. Run -`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing -worker nodes, destroy only the existing `prometheus-stack` Helm release, delete -its retained PVC/PV objects, and recreate the stack on the worker selector when -you intentionally accept losing that monitoring data. A planned monitoring data -migration should be handled as a separate maintenance task with backup, -delete/recreate or storage migration steps, and post-restore checks. - -The older NodePort path is now reserved for special cases such as the local -registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` -support, but app traffic should normally enter through Traefik's MetalLB -address instead of per-app NodePorts. Before applying the edge stack, Jeannie -configures a Tailscale subnet router so the OCI edge can route to the Traefik -LoadBalancer address. The default route advertiser is the Debian control-plane -host because it can reach both Tailscale and the MetalLB address; set -`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. - -The cluster stack still supports worker-based subnet-route advertisement when -that is useful: - -```hcl -metallb = { - enabled = true - repository = "https://metallb.github.io/metallb" - version = "0.16.0" - namespace = "metallb-system" - address_pool = ["192.168.100.240-192.168.100.240"] - l2_advertisement_enabled = true - pool_name = "homelab-lan" -} - -traefik = { - enabled = true - repository = "https://helm.traefik.io/traefik" - chart_version = "40.2.0" - namespace = "traefik" - load_balancer_ip = "192.168.100.240" - ingress_class = "traefik" -} - -tailscale_subnet_routes = { - enabled = true - worker_key = "raspberrypi" - routes = ["192.168.100.0/24"] -} -``` - -DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for -the service hostnames land on the same edge host. For direct LAN testing, point -LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at -Traefik's address: - -```text -192.168.100.240 lab2025.duckdns.org -192.168.100.240 demos.lab2025.duckdns.org -192.168.100.240 n8n.lab2025.duckdns.org -192.168.100.240 prowlarr.lab2025.duckdns.org -192.168.100.240 sonarr.lab2025.duckdns.org -192.168.100.240 radarr.lab2025.duckdns.org -192.168.100.240 qbittorrent.lab2025.duckdns.org -192.168.100.240 kapowarr.lab2025.duckdns.org -192.168.100.240 suwayomi.lab2025.duckdns.org -192.168.100.240 maintainerr.lab2025.duckdns.org -192.168.100.240 grafana.lab2025.duckdns.org -192.168.100.240 argocd.lab2025.duckdns.org -``` - -Heimdall is external to Kubernetes. For direct LAN testing, use -`http://192.168.100.73:8082/`; public access still uses -`https://heimdall.lab2025.duckdns.org/` through the OCI edge. - -The edge stack uses Traefik as its backend by default and validates -`http://192.168.100.240:80/` before updating the edge containers. If Tailscale -subnet-route approval is not automatic in the tailnet policy, the edge deploy -will fail clearly instead of silently keeping a broken public path. - -For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of -remote SSH targets when more worker nodes exist. Set it to an empty string for a -single-node rebuild. - -## Adding Platform Tools - -Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. - -## Policy Guardrails - -`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security -policies in `Audit` mode. This gives the lab policy reports for unsafe workload -settings without blocking existing pods during the first rollout. After reports -are clean, individual policies can be promoted to `Enforce` in -`bootstrap/platform/main.tf`. - -`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for -the images built by this repo and pushed to the local registry. The policy -targets `192.168.100.73:30500/php-website:*` and -`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, -and audits whether the image has both: - -- a valid Cosign signature from the homelab signing key -- a signed SPDX SBOM attestation - -The private Cosign key, generated password, and downloaded Cosign binary live in -`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing -identity survives temporary checkout cleanup. Repo-local image state files still -live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses -the persistent Cosign key, publishes the public key into the Kyverno namespace as -`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs -the pushed images, attaches a signed SPDX SBOM predicate, and verifies both -artifacts before refreshing the workload Applications. The policy starts in -`Audit` mode so an existing live deployment can recover while the current images -are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in -`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to -`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to -manage the key password externally; otherwise the Debian runner creates -`cosign/cosign.password` under the homelab state directory for non-interactive -runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override -those paths. - -## DNS Cache - -`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with -`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on -`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the -rollout compatible with the current kube-proxy iptables path without rewriting -kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service -CIDR or upstream DNS servers change. - -## MetalLB - -MetalLB is present in `bootstrap/platform` but disabled by default. Enable it -only after reserving a LAN IP range outside DHCP and outside any future OpenWrt -LAN pool: - -```bash -export TF_VAR_metallb='{ - enabled = true - repository = "https://metallb.github.io/metallb" - version = "0.16.0" - namespace = "metallb-system" - address_pool = ["192.168.100.240-192.168.100.250"] - l2_advertisement_enabled = true - pool_name = "homelab-lan" -}' -``` - -Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services -such as the website, demos, and n8n should be `ClusterIP` services behind -Ingress objects. The local registry remains a `NodePort` because the cluster -nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; -they run on the Debian Docker host. - -## Secrets - -Use SOPS with age for secrets that need to live in Git. The Debian host -bootstrap installs `age` and `sops`; then run: - -```bash -./{{ main_script }} secrets-init -./{{ main_script }} secrets-check -``` - -`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders -`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. -Commit `.sops.yaml`, but keep the private age key outside the repo. Operational -notes are in `docs/secrets.md`. - -## Edge Services - -The OCI jump box runs the public edge path: - -```text -nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP -``` - -Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. -Validate it before applying with: - -```bash -./{{ main_script }} tailnet-policy-check -``` - -The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and -deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in -`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` -file when the public host, SSH key, server name, backend Tailscale IP, or -Traefik backend address changes. - -The `/git/` route is intentionally different from the Kubernetes app routes: it -proxies to Gitea on the Debian host through Tailscale instead of Traefik. This -keeps public read-only source browsing available even when the cluster has been -destroyed. - -For the main website, nginx terminates TLS, serves cached HTML and static -assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS -sessions, and keeps upstream connections open to the Traefik backend. The -application side marks pre-rendered HTML as -`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS -under the stronger immutable asset policy. In local measurements, cached -single-resource requests are dominated by the remote TCP/TLS path, while a -reused HTTP/2 asset batch completes in roughly 60 ms after the connection is -established. - -Use the configured `server_name` in the browser, for example -`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser -certificate warning because the trusted certificate is issued for the hostname. - -The edge stack uses HTTP-01 validation and requests one certificate covering -`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains -under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must -be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive -expiry notices, or leave it empty to register without an email. Set -`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. - -Operational runbooks: - -- [Edge failures](docs/runbooks/edge-failures.md) -- [Gitea failures](docs/runbooks/gitea-failures.md) -- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) - -## Adding Apps - -Add Kubernetes manifests under `apps/` and register them in -`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and -self-healing for the app. - -The `heimdall` Argo CD app name is retained as a small route bundle for -Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can -prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes -workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard -itself. - -The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is -exposed at `https://n8n.lab2025.duckdns.org`. It receives -`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing -Debian-host Ollama API for translation prewarming and batch jobs. - -The ARR/media stack is intentionally not deployed by the Kubernetes app -pipeline. It is better suited to a Docker Compose stack on the Debian host with -persistent volumes under `/data`, where large media paths, qBittorrent writes, -and service config directories can survive cluster rebuilds without OpenEBS -node-affinity or PVC migration concerns. The active host-level Compose -definition lives under `infra/arr-stack` and stores persistent data under -`/data/arr`. Historical Kubernetes manifests are archived under -`archive/apps/arr-stack` for migration reference only. - -## Storage - -OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use -retained local PV paths such as `/data/openebs/local/registry`; these paths are -intentionally outside kubeadm reset paths so data can survive cluster -destroy/create cycles. Those critical volumes are declared explicitly as -retained local PVs so a rebuilt cluster binds back to the same host paths -instead of creating fresh directories. - -For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the -standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath -data still lives under `/data/openebs/local`, and larger service data such as -Gitea, Ollama models, and app volumes should stay under `/data`. - -## Gitea - -Gitea is external bootstrap infrastructure. It runs on the Debian host as an -always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a -Kubernetes workload. This keeps Git available when the Kubernetes cluster is -destroyed and rebuilt. - -The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. -Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. - -Public source browsing stays available through -`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users -can view public repositories, so the blog can link to code read-only while -writes still require an authenticated Gitea account. - -## Heimdall - -Heimdall is external bootstrap infrastructure. It runs on the Debian host as an -always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not -as a Kubernetes workload. This keeps the lab navigation page available when the -cluster is destroyed, rebuilding, or unhealthy. - -Deploy or refresh it from the Debian server with: - -```bash -./{{ main_script }} deploy-heimdall -``` - -`./{{ main_script }} up` also runs this step during the independent host -services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path -is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. -Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar -container that upserts repo-managed tiles without deleting manually added ones. - -The Debian bare repo remains the GitOps mirror: - -```text -/home/jv/git-server/my-homelab-configs.git -``` - -Argo CD consumes that Debian mirror through the default `gitops_repo_url`. -Gitea Actions pushes the `main` commit into the mirror before running the -selected deploy command. - -The platform bootstrap registers the Argo CD repository secret and the SSH host -key for the Debian GitOps mirror. If Argo CD reports -`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host -key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, -restart `argocd-repo-server`, and hard-refresh the affected Application. - -Deploy or refresh the external Gitea container from the Debian host with: - -```bash -./{{ main_script }} deploy-gitea -``` - -## Gitea Backups - -`./{{ main_script }} up` installs a Debian-host systemd timer named -`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured -Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the -dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default -retention is 30 days. - -The same install step also creates `homelab-gitea-restore-drill.timer`. The -monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it -to a temporary directory, records a report under -`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It -does not write into the live Gitea data directory. - -Run a manual backup from the Debian server with: - -```bash -./{{ main_script }} backup-gitea -``` - -Run the restore drill manually with: - -```bash -./{{ main_script }} drill-gitea-restore -./{{ main_script }} drill-restore -./{{ main_script }} drill-pihole-restore -``` - -`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed -config inputs, the latest OpenTofu state backup archive when present, and the -GitOps rebuild path. It does not replace running services or mutate Kubernetes. - -Useful checks: - -```bash -./{{ main_script }} backup-status -systemctl list-timers homelab-gitea-backup.timer -systemctl list-timers homelab-gitea-restore-drill.timer -sudo systemctl start homelab-gitea-backup.service -ls -lh /home/jv/backups/gitea -ls -lh /home/jv/backups/gitea-restore-drills -``` - -## Gitea Actions - -This repo includes a Gitea Actions workflow at -`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and -`main`, and deploys only from `main`. That keeps the promotion path simple: - -```text -dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync -``` - -The workflow targets a repository-scoped Debian host runner with the label -`homelab-debian`. - -Run the same validation locally before promoting: - -```bash -./{{ main_script }} validate -``` - -## Identity and access audit - -Run a read-only access inventory from the Debian server with: - -```bash -./{{ main_script }} access-audit -``` - -The audit reports configured SSH targets and key file modes, Git/Gitea remotes, -Tailscale ACL policy validation, current Kubernetes authorization, broad -cluster-admin bindings, automation service accounts, and Gitea runner status. -It does not create users, tokens, kubeconfigs, or keys. Use it to move toward -separate identities: - -- admin kubeconfig only on the Debian control host -- read-only kubeconfig for dashboards and audits -- separate service accounts for automation -- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge -- repo-managed Tailscale ACLs - -The read-only Kubernetes identity is managed in `apps/access-control`. After -that app syncs, generate a separate read-only kubeconfig on the Debian host: - -```bash -./{{ main_script }} kubeconfig-readonly -``` - -The workflow only blocks automatic deploy for external Gitea service -changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, -`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` -functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by -default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already -has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; -if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no -kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. - -Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` -on the runner to force one path. - -`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key -with the Gitea repository and switches the Debian working copy's `gitea` remote -to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key -is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a -different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH -access unchanged. The Actions deploy job uses the checked-out Actions workspace -as the source commit, updates the first available persistent checkout from -`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or -`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the -Actions workspace. It does not need SSH read access back to Gitea. - -Enable Actions for the repository in Gitea, then create a repository-level runner -token from: - -```text -https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners -``` - -Register and start the Debian runner from the Debian server: - -```bash -cd ~/my-homelab-configs -GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner -``` - -The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and -uses a host label instead of a Docker job container because deployment needs the -Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. - -The deployment job is non-interactive. User `jv` must be able to run `sudo -n -true` on the Debian host for deployment commands that require sudo. - -Useful checks: - -```bash -systemctl status homelab-gitea-runner.service -journalctl -u homelab-gitea-runner.service -n 100 --no-pager -``` - -## Renovate - -`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu -providers, Helm chart versions, and the pinned tools used by the Gitea Actions -workflow. Renovate should open reviewable update branches or PRs only; it must -not auto-merge infrastructure changes. Keep app-only dependency updates on the -normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server -for platform or provisioning updates. - -## Destructive Rebuilds - -`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico -links, iptables rules, and local OpenTofu state. It does not delete retained data -under `/data/openebs/local`. - -For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH -targets. It defaults to an empty string so worker nodes are not cleaned unless -you explicitly include them. - -## Website App - -The website is a PHP app under `apps/website`. It includes a home page, CV page, -blog page, and demos page, plus a lightweight translation flow backed by Redis, -n8n, and Ollama. -Static language files live in `apps/website/lang`; `en.php` and `nah.php` are -curated source files, with the Nahuatl home page intentionally biased toward as -many Nahuatl words as possible while keeping technical terms understandable. -Unsupported browser languages use the same-origin `/translate.php` endpoint, -which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the -browser never calls the private Ollama IP directly. Redis caches per-string -translation results in the `website-production` namespace, while n8n is -available for translation prewarming and batch workflow jobs. The default model -is the custom `website-translator` Ollama model defined in -`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved -through `save_lang.php` on the website PVC, and `translate.php` emits structured -logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. - -Create or refresh the Ollama model on the Debian server before deploying a -website image that points at it: - -```bash -./{{ main_script }} website-translation-model -``` - -Ollama must also listen on the Debian host LAN address so Kubernetes pods on -other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: - -```bash -./{{ main_script }} website-ollama-listen -``` - -The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` -is true: it installs the service, stores models under `/data/ollama/models`, -binds the API to the configured LAN listener, and pulls the lightweight -`ai_gateway.model`. On an existing host, apply just this setup with: - -```bash -./{{ main_script }} ollama-setup -``` - -`{{ main_script }}` also has a backstage local AI helper for doctor commands. -It is not a separate CLI action; when `ai_gateway.enabled` is true in -`homelab.yml`, doctor commands try the configured Ollama model and silently -skip the helper if Ollama is down. The default lightweight model is: - -```bash -ollama pull qwen2.5:0.5b -``` - -Disable it for low-CPU sessions with: - -```bash -LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge -``` - -Build the local homelab knowledge index separately from the main deployment: - -```bash -./{{ main_script }} ai-index -./{{ main_script }} ai-check -./{{ main_script }} ask "how do I check edge to Gitea?" -``` - -The index is built from the non-secret source list in -`infra/ai/knowledge-sources.txt`, skips paths in -`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. -Doctor commands use it as extra context when it exists, but infrastructure -deployment does not depend on it. -Use `ask` as the runbook and command finder. It returns the closest indexed -docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after -retrieval. - -The CV page has two client-side presentation modes: - -- `Elegant`: dark, minimal, terminal-inspired styling with a square profile - image and light green console text. -- `Fancy`: centered circular profile image, cursive orbit text, and a - cursor-following portrait rotation effect. - -The Demos page is a catalog in the PHP website. The actual demo applications are -served from a separate `demos-static` artifact under `apps/demos-static` and are -published through the `demos-static` Argo CD application. Public traffic reaches -them through the edge path at `/demo-apps/`. - -`./{{ main_script }} up` builds and pushes two independent images: - -- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD - as a Kustomize image override -- `demos-static:latest` from `apps/demos-static` -- Cosign signatures and signed SPDX SBOM attestations for both pushed images - -The website manifest keeps the stable base image name `php-website:bootstrap`. -During bootstrap, `{{ main_script }}` hashes `apps/website`, builds -`/php-website:src-`, exports that exact reference through -`TF_VAR_website_image_ref`, and the Argo CD Application applies it through -Kustomize. This keeps the GitOps source generic while the deployed image remains -immutable. The Kyverno supply-chain policy mutates admitted pods to the verified -image digest, so the workload runs the same digest that was signed and attested. - -After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, -for example `192.168.100.73:30500/php-website:src-...`. If it still shows -`php-website:latest`, Argo CD has not rendered the current Application source. -Check the `website-production` Application source, sync status, and repository -access before restarting pods. - -The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs -private, browser-only image compression and conversion using native Canvas APIs. -Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled -to WebAssembly with a TypeScript UI so the codec work stays fast and still -avoids backend uploads. - -The demos are designed to be local-first so the current cluster can serve them -from any Linux app node without turning either pod into an application server. -The website pod serves the portfolio shell and the `demos-static` pod serves -static demo bundles; CPU-heavy work runs in the visitor's browser. Because the -deployments can run on either Debian or ARM workers, avoid bundling large ML -models, server-side WebSocket probes, or backend video transcoders into either -image. If those demos become production-grade, lazy load model assets in the -browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 -Plus. - -Current demo inventory: - -- Client-side media cruncher: image conversion/compression with Canvas; future - Rust/Wasm codec path for video. -- Internet quality visualizer: live Canvas graph for latency, jitter, and - stability using same-origin browser probes; a dedicated WebSocket echo endpoint - would be the production version. -- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and - local text-log filtering. -- Architecture simulator: click-driven load, crash, and auto-scale simulation. -- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB - conversions. -- Privacy-first redactor: local image redaction prototype; future - onnxruntime-web plus quantized YOLO or face model path. -- Local sentiment sandbox: lightweight local sentiment, keyword, and summary - prototype; future Transformers.js/ONNX path. -- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, - and retraining. - -The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. -Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by -the app image build and with the image globs in -`apps/supply-chain-policy/local-registry-image-policy.yaml`. - -Keep the `.terraform.lock.hcl` files committed. They pin provider selections and -make bootstrap behavior reproducible across nodes and rebuilds. +Run `./{{ main_script }} up` and `./{{ main_script }} nuke` only from the Debian homelab server. diff --git a/docs/architecture-diagram.html b/docs/architecture-diagram.html new file mode 100644 index 0000000..a2c5a41 --- /dev/null +++ b/docs/architecture-diagram.html @@ -0,0 +1,221 @@ + + + + + + Homelab Infrastructure Architecture + + + + +
+
+
+
+

Homelab Infrastructure Architecture

+
+

Hybrid K8s + Docker Compose + OCI Edge Bridge

+
+ +
+ + + + + + + + + + + + + + + + Users + Public Internet + + + + OCI Edge (Jump Box) + + + Nginx + TLS / Gzip / H2 + + + HAProxy + L7 Routing + + + Varnish/Squid + Edge Cache + + + + HTTPS + + + + + + Tailscale Mesh + + + + Homelab LAN (192.168.100.0/24) + + + + debian:control-plane + + + Gitea + GitOps Source + + + Heimdall + App Dashboard + + + Ollama + Local LLM API + + + + Kubernetes Cluster + + + Traefik + Ingress (MetalLB) + + + + Web Apps + • Website (PHP) + • Demos Static + • n8n + + + + Platform + • Argo CD + • Prometheus/Grafana + + + + Workers + • Pimox VMs + • Raspberry Pi + + + + Storage + OpenEBS Hostpath + + + + + + + + + + App Path + + + GitOps/SSH + + + + Ollama API + + + Legend + + Frontend/App + + Backend/Platform + + OCI Cloud + + Storage/DB + + AI/Security + +
+ +
+
+
+
+

Edge Gateway

+
+
    +
  • • TLS termination at OCI Edge
    • +
  • • Varnish cache for static HTML
    • +
  • • Tailscale tunnel to Homelab LAN
    • +
+
+
+
+
+

K8s Core

+
+
    +
  • • Traefik L7 Ingress via MetalLB
    • +
  • • Argo CD GitOps reconciliation
    • +
  • • Kyverno supply-chain enforcement
    • +
+
+
+
+
+

AI & Data

+
+
    +
  • • Ollama local LLM on Debian host
    • +
  • • Redis translation cache (Website)
    • +
  • • OpenEBS local PV retention
    • +
+
+
+ +

+ Homelab Infrastructure • v1.0.0 • Generated by Hermes Agent +

+
+ + diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..deeff9b --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,32 @@ +# Architecture Overview + +![Infrastructure Diagram](./architecture-diagram.html) +*(Note: Since this is an HTML file, click the link below to open it in your browser)* + +[👉 Open Interactive Architecture Diagram](./architecture-diagram.html) + +## System Design + +This homelab utilizes a hybrid architecture combining a high-performance K8s cluster for scalable workloads and a stable Docker-Compose/Host environment for core infrastructure services. + +### 1. The Edge Path (OCI $\rightarrow$ LAN) +Traffic enters via the OCI Edge bridge to minimize exposure of the home IP and provide low-latency caching. +- **Ingress:** Nginx handles TLS termination and HTTP/2. +- **Routing:** HAProxy manages L7 routing to the internal mesh. +- **Caching:** Varnish/Squid provides an edge cache for static content. +- **Transport:** A Tailscale mesh tunnel connects the OCI Edge to the internal homelab LAN. + +### 2. The Core Infrastructure (LAN) +The internal network is centered around a Debian control-plane and a mix of ARM64 worker nodes. +- **Host Services:** Core GitOps (Gitea), Dashboard (Heimdall), and Local AI (Ollama) run directly on the Debian host for maximum reliability. +- **K8s Cluster:** Managed via `jeannie` (OpenTofu + ArgoCD). + - **Ingress:** Traefik provides L7 routing via MetalLB. + - **Workloads:** Segmented into `Web Apps` (PHP, Static) and `Platform Services` (ArgoCD, Prometheus). + - **Workers:** A mix of Pimox-virtualized VMs and Raspberry Pi nodes. +- **Storage:** Local persistence is managed via OpenEBS Hostpath. + +### 3. AI Integration Flow +AI capabilities are decoupled from the K8s lifecycle to utilize host-level GPU/CPU resources efficiently. +- **API Layer:** Ollama runs as a host service. +- **Consumption:** K8s-based applications communicate with the Ollama API over the internal LAN. +- **Memory:** Redis (transitioning to Valkey) provides a translation and session cache for AI-driven web features. diff --git a/docs/templates/ai-ops.md.tmpl b/docs/templates/ai-ops.md.tmpl new file mode 100644 index 0000000..ad53673 --- /dev/null +++ b/docs/templates/ai-ops.md.tmpl @@ -0,0 +1,15322 @@ +# Jeannie AI & Operations + +## Implementation Language Policy + +Jeannie keeps `./{{ main_script }}` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./{{ main_script }} doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./{{ main_script }} ai-evals run +``` + +Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./{{ main_script }} doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./{{ main_script }} ai-evals run +``` + +Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./{{ main_script }} ai-evals run +``` + +Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + diff --git a/docs/templates/app-ops.md.tmpl b/docs/templates/app-ops.md.tmpl new file mode 100644 index 0000000..60b958d --- /dev/null +++ b/docs/templates/app-ops.md.tmpl @@ -0,0 +1,2624 @@ +# Application Operations + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. diff --git a/docs/templates/architecture.md.tmpl b/docs/templates/architecture.md.tmpl new file mode 100644 index 0000000..6dc83fe --- /dev/null +++ b/docs/templates/architecture.md.tmpl @@ -0,0 +1,6172 @@ +# Homelab Architecture & Infrastructure + +## Architecture + +The lab is intentionally small but production-shaped: + +- a Debian amd64 host runs the kubeadm control plane and local deployment tools +- a Raspberry Pi arm64 node runs selected workloads +- the Debian host also runs the always-on Gitea Docker service outside + Kubernetes +- the Debian host keeps a bare GitOps mirror under + `/home/jv/git-server/my-homelab-configs.git` +- a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker + templates +- OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge +- Argo CD continuously reconciles Kubernetes manifests from this repo +- a local registry stores the website and demos images built for the worker + architecture +- the website image keeps Apache as the public server, uses PHP-FPM with + OPcache for dynamic endpoints, and pre-renders read-heavy pages to static + HTML during the image build +- SOPS with age is the committed secret-management path for future encrypted + Kubernetes secrets +- local Jeannie/RAG eval cases exercise troubleshooting explanations and + retrieval grounding without touching live infrastructure +- an OCI jump box provides the public edge path back into the homelab over + Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream + keepalive + +## Inventory + +The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, +Tailscale IPs, public hostnames, ports, storage paths, and service ownership +there before copying values into scripts, OpenTofu variables, runbooks, or +service docs. Secrets and tokens do not belong in that file. + +`./{{ main_script }}` reads `homelab.yml` at startup and exports matching OpenTofu +variables for the edge, provisioning, cluster, platform, and apps stacks. This +keeps values such as the Debian LAN IP, Debian Tailscale IP, RPi LAN IP, OCI IP, +Traefik MetalLB IP, and domain name from drifting across scripts and templates. +`scripts/render-docs` also renders [docs/service-catalog.md](docs/service-catalog.md) +and the static service catalog page at +`apps/demos-static/public/homelab-catalog/index.html` from the same inventory. + +Run `./{{ main_script }} up` and `./{{ main_script }} nuke` only from the Debian homelab server. The +script intentionally refuses to run from non-Debian machines so a laptop cannot +accidentally modify the cluster. + +## Flow + +1. `bootstrap/provisioning` + - prepares a Debian server as a PXE and preseed service for arm64 VMs + - serves Debian 13 arm64 netboot assets through TFTP and HTTP + - exposes a PXE rescue menu and HTTP rescue page for manual repair paths + - creates a golden image install path with Kubernetes, containerd, + qemu-guest-agent, cloud-init, and storage client packages ready + - is driven by `./{{ main_script }} up` when Pimox is reachable, without changing + Orange Pi host networking + +2. `bootstrap/cluster` + - creates the kubeadm control plane on the Debian amd64 node + - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes + - configures Calico-compatible pod CIDR + - configures containerd to pull from the in-cluster NodePort registry + - creates retained host directories under `/data/openebs/local` + +3. `bootstrap/platform` + - installs a minimal Calico deployment through the Tigera operator + - installs NodeLocal DNSCache for node-local DNS query caching + - installs MetalLB for LAN `LoadBalancer` services + - installs Traefik as the single Kubernetes ingress entry point + - installs OpenEBS + - creates `openebs-hostpath-retain` + - installs Argo CD + - installs Kyverno with audit-first baseline Pod Security policies and + homelab image signature/SBOM admission controls + - registers the private GitOps repo without storing the SSH private key in + Terraform state + +4. `bootstrap/apps` + - registers Argo CD Applications from the `applications` map + - passes the website image produced by the build step into Argo CD as a + Kustomize image override + - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM + listens on loopback inside the container, OPcache is enabled, and + pre-rendered HTML handles the read-heavy pages + - default apps are `container-registry`, `website-production`, + `demos-static`, the `heimdall` route bundle, `n8n`, and + `supply-chain-policy` + +5. `bootstrap/edge` + - connects to the OCI jump box + - uploads nginx, HAProxy, Varnish, and Squid configs + - obtains and renews Let's Encrypt certificates for the configured hostname + - runs the edge cache/proxy chain with Docker Compose + - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream + keepalive, and cache headers for the public website path + +## Prerequisites + +On the Debian host: + +- OpenTofu +- Docker with Buildx +- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes + minor, currently `v1.36` +- SSH access to worker nodes +- SSH access to the OCI edge host +- Go toolchain for Jeannie core helpers; `./{{ main_script }} up` installs + `golang-go` on the Debian host if it is missing +- enough persistent storage for `/data/openebs/local` and Docker's data root + +After a clean Debian reinstall, bootstrap the host package set and desktop +configuration first: + +```bash +sudo apt-get update +sudo apt-get install -y git ansible sudo +git clone /home/jv/my-homelab-configs +cd /home/jv/my-homelab-configs +ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml +``` + +If you kept the private app-config archive generated from the old Debian host, +pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. +Details are in `bootstrap/host/README.md`. + +The default kubeconfig path is `/home/jv/.kube/config`. Override it with +`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. + +## Implementation Language Policy + +Jeannie keeps `./{{ main_script }}` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./{{ main_script }} doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./{{ main_script }} ai-evals run +``` + +Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Flow + +1. `bootstrap/provisioning` + - prepares a Debian server as a PXE and preseed service for arm64 VMs + - serves Debian 13 arm64 netboot assets through TFTP and HTTP + - exposes a PXE rescue menu and HTTP rescue page for manual repair paths + - creates a golden image install path with Kubernetes, containerd, + qemu-guest-agent, cloud-init, and storage client packages ready + - is driven by `./{{ main_script }} up` when Pimox is reachable, without changing + Orange Pi host networking + +2. `bootstrap/cluster` + - creates the kubeadm control plane on the Debian amd64 node + - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes + - configures Calico-compatible pod CIDR + - configures containerd to pull from the in-cluster NodePort registry + - creates retained host directories under `/data/openebs/local` + +3. `bootstrap/platform` + - installs a minimal Calico deployment through the Tigera operator + - installs NodeLocal DNSCache for node-local DNS query caching + - installs MetalLB for LAN `LoadBalancer` services + - installs Traefik as the single Kubernetes ingress entry point + - installs OpenEBS + - creates `openebs-hostpath-retain` + - installs Argo CD + - installs Kyverno with audit-first baseline Pod Security policies and + homelab image signature/SBOM admission controls + - registers the private GitOps repo without storing the SSH private key in + Terraform state + +4. `bootstrap/apps` + - registers Argo CD Applications from the `applications` map + - passes the website image produced by the build step into Argo CD as a + Kustomize image override + - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM + listens on loopback inside the container, OPcache is enabled, and + pre-rendered HTML handles the read-heavy pages + - default apps are `container-registry`, `website-production`, + `demos-static`, the `heimdall` route bundle, `n8n`, and + `supply-chain-policy` + +5. `bootstrap/edge` + - connects to the OCI jump box + - uploads nginx, HAProxy, Varnish, and Squid configs + - obtains and renews Let's Encrypt certificates for the configured hostname + - runs the edge cache/proxy chain with Docker Compose + - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream + keepalive, and cache headers for the public website path + +## Prerequisites + +On the Debian host: + +- OpenTofu +- Docker with Buildx +- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes + minor, currently `v1.36` +- SSH access to worker nodes +- SSH access to the OCI edge host +- Go toolchain for Jeannie core helpers; `./{{ main_script }} up` installs + `golang-go` on the Debian host if it is missing +- enough persistent storage for `/data/openebs/local` and Docker's data root + +After a clean Debian reinstall, bootstrap the host package set and desktop +configuration first: + +```bash +sudo apt-get update +sudo apt-get install -y git ansible sudo +git clone /home/jv/my-homelab-configs +cd /home/jv/my-homelab-configs +ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml +``` + +If you kept the private app-config archive generated from the old Debian host, +pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. +Details are in `bootstrap/host/README.md`. + +The default kubeconfig path is `/home/jv/.kube/config`. Override it with +`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. + +## Implementation Language Policy + +Jeannie keeps `./{{ main_script }}` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./{{ main_script }} doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./{{ main_script }} ai-evals run +``` + +Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + diff --git a/docs/templates/deployment.md.tmpl b/docs/templates/deployment.md.tmpl new file mode 100644 index 0000000..f0fabf0 --- /dev/null +++ b/docs/templates/deployment.md.tmpl @@ -0,0 +1,3605 @@ +# Homelab Deployment Guide + +## Prerequisites + +On the Debian host: + +- OpenTofu +- Docker with Buildx +- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes + minor, currently `v1.36` +- SSH access to worker nodes +- SSH access to the OCI edge host +- Go toolchain for Jeannie core helpers; `./{{ main_script }} up` installs + `golang-go` on the Debian host if it is missing +- enough persistent storage for `/data/openebs/local` and Docker's data root + +After a clean Debian reinstall, bootstrap the host package set and desktop +configuration first: + +```bash +sudo apt-get update +sudo apt-get install -y git ansible sudo +git clone /home/jv/my-homelab-configs +cd /home/jv/my-homelab-configs +ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml +``` + +If you kept the private app-config archive generated from the old Debian host, +pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. +Details are in `bootstrap/host/README.md`. + +The default kubeconfig path is `/home/jv/.kube/config`. Override it with +`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. + +## Implementation Language Policy + +Jeannie keeps `./{{ main_script }}` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./{{ main_script }} doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./{{ main_script }} ai-evals run +``` + +Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./{{ main_script }} prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./{{ main_script }} incident replay +./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./{{ main_script }} safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./{{ main_script }} impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./{{ main_script }} agent-sandbox check "./{{ main_script }} status" +./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./{{ main_script }} ai-scheduler recommend small-rag +./{{ main_script }} ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./{{ main_script }} ask --citations "how do I recover the cluster?" +./{{ main_script }} ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./{{ main_script }} red-blue plan +./{{ main_script }} red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./{{ main_script }} promote plan +./{{ main_script }} promote validate +./{{ main_script }} promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./{{ main_script }} status --heal-plan +./{{ main_script }} status --heal +./{{ main_script }} heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./{{ main_script }} workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./{{ main_script }} model-observe prompts +./{{ main_script }} model-observe run --offline +./{{ main_script }} model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./{{ main_script }} up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./{{ main_script }} workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` +runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./{{ main_script }} preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./{{ main_script }} fix-debian-docker-root +``` + +When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./{{ main_script }} help +./{{ main_script }} status +./{{ main_script }} capacity +./{{ main_script }} capacity-advisor +./{{ main_script }} capacity-limits +./{{ main_script }} recover-plan +./{{ main_script }} recover-power --dry-run +./{{ main_script }} impact --since HEAD~1 +./{{ main_script }} review-last-change +./{{ main_script }} scorecard +./{{ main_script }} explain scorecard +./{{ main_script }} gitops-status +./{{ main_script }} cert-check +./{{ main_script }} grafana-dashboards list +./{{ main_script }} release-snapshot +./{{ main_script }} backup-status +./{{ main_script }} synthetic-checks +./{{ main_script }} resource-budget +./{{ main_script }} control-plane status +./{{ main_script }} artifact-cache status +./{{ main_script }} blockchain-devnet status +./{{ main_script }} blockchain-test +./{{ main_script }} blockchain-wallet instructions +./{{ main_script }} golden-ledger check +./{{ main_script }} route-inventory +./{{ main_script }} workers list +./{{ main_script }} change-journal list +./{{ main_script }} map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./{{ main_script }} status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./{{ main_script }} explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./{{ main_script }} explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions +them through the platform stack; `./{{ main_script }} grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./{{ main_script }} resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./{{ main_script }} doctor-edge +./{{ main_script }} doctor-gitea +./{{ main_script }} doctor-rpi +./{{ main_script }} doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./{{ main_script }} rpi-services +``` + +`./{{ main_script }} up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./{{ main_script }} apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./{{ main_script }} up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./{{ main_script }} rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./{{ main_script }} stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./{{ main_script }} start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./{{ main_script }} secrets-init +./{{ main_script }} secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./{{ main_script }} tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./{{ main_script }} deploy-heimdall +``` + +`./{{ main_script }} up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./{{ main_script }} deploy-gitea +``` + +## Gitea Backups + +`./{{ main_script }} up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./{{ main_script }} backup-gitea +``` + +Run the restore drill manually with: + +```bash +./{{ main_script }} drill-gitea-restore +./{{ main_script }} drill-restore +./{{ main_script }} drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./{{ main_script }} backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./{{ main_script }} validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./{{ main_script }} access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./{{ main_script }} kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; +if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no +kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./{{ main_script }} website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./{{ main_script }} website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./{{ main_script }} ollama-setup +``` + +`{{ main_script }}` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./{{ main_script }} ai-index +./{{ main_script }} ai-check +./{{ main_script }} ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./{{ main_script }} up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `{{ main_script }}` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Destructive Rebuilds + +`./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + diff --git a/scripts/render-docs b/scripts/render-docs index ae1dd7b..aedb23b 100755 --- a/scripts/render-docs +++ b/scripts/render-docs @@ -6,6 +6,8 @@ REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" INVENTORY_FILE="${REPO_ROOT}/homelab.yml" README_TEMPLATE="${REPO_ROOT}/README.md.tmpl" README_OUTPUT="${REPO_ROOT}/README.md" +TEMPLATES_DIR="${REPO_ROOT}/docs/templates" +DOCS_DIR="${REPO_ROOT}/docs" usage() { cat <<'USAGE' @@ -57,22 +59,22 @@ print(value) PY } -render_readme() { - local main_script="$1" +render_template() { + local template_path="$1" + local output_path="$2" + local main_script="$3" - if grep -Eo '{{[[:space:]]*[A-Za-z_][A-Za-z0-9_]*[[:space:]]*}}' "${README_TEMPLATE}" | - grep -Evq '{{[[:space:]]*main_script[[:space:]]*}}'; then - printf 'Unsupported template variable in %s\n' "${README_TEMPLATE}" >&2 + if [[ ! -f "$template_path" ]]; then return 1 fi - sed "s/{{[[:space:]]*main_script[[:space:]]*}}/${main_script}/g" "${README_TEMPLATE}" + sed "s/{{[[:space:]]*main_script[[:space:]]*}}/${main_script}/g" "$template_path" > "$output_path" } main() { local mode="render" local main_script - local rendered + local rendered_tmp case "${1:-}" in "") @@ -94,12 +96,6 @@ main() { printf 'Missing homelab inventory: %s\n' "${INVENTORY_FILE}" >&2 return 1 fi - if [[ ! -s "${README_TEMPLATE}" ]]; then - printf 'Missing README template: %s\n' "${README_TEMPLATE}" >&2 - return 1 - fi - - "${REPO_ROOT}/scripts/validate-homelab-inventory" "${INVENTORY_FILE}" main_script="$(read_inventory_value metadata.main_script)" if [[ -z "${main_script}" ]]; then @@ -107,25 +103,55 @@ main() { return 1 fi - rendered="$(mktemp)" - render_readme "${main_script}" >"${rendered}" + # 1. Render README + rendered_tmp=$(mktemp) + render_template "${README_TEMPLATE}" "$rendered_tmp" "${main_script}" if [[ "${mode}" == "check" ]]; then - "${REPO_ROOT}/scripts/render-service-catalog" --check - if ! cmp -s "${rendered}" "${README_OUTPUT}"; then - printf 'Generated README.md is stale. Run scripts/render-docs.\n' >&2 - diff -u "${README_OUTPUT}" "${rendered}" || true - rm -f "${rendered}" + if ! cmp -s "$rendered_tmp" "${README_OUTPUT}"; then + printf 'Generated README.md is stale.\n' >&2 + diff -u "${README_OUTPUT}" "$rendered_tmp" || true + rm -f "$rendered_tmp" return 1 fi - printf 'docs are current\n' - rm -f "${rendered}" - return 0 + else + mv "$rendered_tmp" "${README_OUTPUT}" + fi + rm -f "$rendered_tmp" + + # 2. Render Modular Docs + for tmpl in "${TEMPLATES_DIR}"/*.tmpl; do + [[ -e "$tmpl" ]] || continue + local filename=$(basename "$tmpl" .tmpl) + local output_file="${DOCS_DIR}/${filename}.md" + + if [[ "${mode}" == "check" ]]; then + rendered_tmp=$(mktemp) + render_template "$tmpl" "$rendered_tmp" "${main_script}" + if ! cmp -s "$rendered_tmp" "$output_file"; then + printf 'Generated %s is stale.\n' "$output_file" >&2 + diff -u "$output_file" "$rendered_tmp" || true + rm -f "$rendered_tmp" + return 1 + fi + rm -f "$rendered_tmp" + else + render_template "$tmpl" "$output_file" "${main_script}" + fi + done + + # 3. Render Service Catalog + if [[ "${mode}" == "check" ]]; then + "${REPO_ROOT}/scripts/render-service-catalog" --check + else + "${REPO_ROOT}/scripts/render-service-catalog" fi - mv "${rendered}" "${README_OUTPUT}" - "${REPO_ROOT}/scripts/render-service-catalog" - printf 'Rendered %s from %s\n' "${README_OUTPUT}" "${README_TEMPLATE}" + if [[ "${mode}" == "render" ]]; then + printf 'Successfully rendered all documentation from templates.\n' + else + printf 'All docs are current.\n' + fi } main "$@"