From cafe804d7192ab63c4f7c94759158dec20f0f6e3 Mon Sep 17 00:00:00 2001 From: jv Date: Wed, 16 Sep 2026 22:17:52 -0500 Subject: [PATCH] Fix render script filename extension bug --- docs/ai-ops.md | 15322 +++++++++++++++++++++++++++++++++++++++++ docs/app-ops.md | 2624 +++++++ docs/architecture.md | 6190 ++++++++++++++++- docs/deployment.md | 3605 ++++++++++ scripts/render-docs | 2 +- 5 files changed, 27717 insertions(+), 26 deletions(-) create mode 100644 docs/ai-ops.md create mode 100644 docs/app-ops.md create mode 100644 docs/deployment.md diff --git a/docs/ai-ops.md b/docs/ai-ops.md new file mode 100644 index 0000000..105ae9a --- /dev/null +++ b/docs/ai-ops.md @@ -0,0 +1,15322 @@ +# Jeannie AI & Operations + +## Implementation Language Policy + +Jeannie keeps `./jeannie` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./jeannie doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./jeannie ai-evals run +``` + +Use `./jeannie ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./jeannie doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./jeannie ai-evals run +``` + +Use `./jeannie ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./jeannie ai-evals run +``` + +Use `./jeannie ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + diff --git a/docs/app-ops.md b/docs/app-ops.md new file mode 100644 index 0000000..2f128c0 --- /dev/null +++ b/docs/app-ops.md @@ -0,0 +1,2624 @@ +# Application Operations + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. diff --git a/docs/architecture.md b/docs/architecture.md index deeff9b..9db2f29 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,32 +1,6172 @@ -# Architecture Overview +# Homelab Architecture & Infrastructure -![Infrastructure Diagram](./architecture-diagram.html) -*(Note: Since this is an HTML file, click the link below to open it in your browser)* +## Architecture -[👉 Open Interactive Architecture Diagram](./architecture-diagram.html) +The lab is intentionally small but production-shaped: -## System Design +- a Debian amd64 host runs the kubeadm control plane and local deployment tools +- a Raspberry Pi arm64 node runs selected workloads +- the Debian host also runs the always-on Gitea Docker service outside + Kubernetes +- the Debian host keeps a bare GitOps mirror under + `/home/jv/git-server/my-homelab-configs.git` +- a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker + templates +- OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge +- Argo CD continuously reconciles Kubernetes manifests from this repo +- a local registry stores the website and demos images built for the worker + architecture +- the website image keeps Apache as the public server, uses PHP-FPM with + OPcache for dynamic endpoints, and pre-renders read-heavy pages to static + HTML during the image build +- SOPS with age is the committed secret-management path for future encrypted + Kubernetes secrets +- local Jeannie/RAG eval cases exercise troubleshooting explanations and + retrieval grounding without touching live infrastructure +- an OCI jump box provides the public edge path back into the homelab over + Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream + keepalive -This homelab utilizes a hybrid architecture combining a high-performance K8s cluster for scalable workloads and a stable Docker-Compose/Host environment for core infrastructure services. +## Inventory -### 1. The Edge Path (OCI $\rightarrow$ LAN) -Traffic enters via the OCI Edge bridge to minimize exposure of the home IP and provide low-latency caching. -- **Ingress:** Nginx handles TLS termination and HTTP/2. -- **Routing:** HAProxy manages L7 routing to the internal mesh. -- **Caching:** Varnish/Squid provides an edge cache for static content. -- **Transport:** A Tailscale mesh tunnel connects the OCI Edge to the internal homelab LAN. +The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, +Tailscale IPs, public hostnames, ports, storage paths, and service ownership +there before copying values into scripts, OpenTofu variables, runbooks, or +service docs. Secrets and tokens do not belong in that file. -### 2. The Core Infrastructure (LAN) -The internal network is centered around a Debian control-plane and a mix of ARM64 worker nodes. -- **Host Services:** Core GitOps (Gitea), Dashboard (Heimdall), and Local AI (Ollama) run directly on the Debian host for maximum reliability. -- **K8s Cluster:** Managed via `jeannie` (OpenTofu + ArgoCD). - - **Ingress:** Traefik provides L7 routing via MetalLB. - - **Workloads:** Segmented into `Web Apps` (PHP, Static) and `Platform Services` (ArgoCD, Prometheus). - - **Workers:** A mix of Pimox-virtualized VMs and Raspberry Pi nodes. -- **Storage:** Local persistence is managed via OpenEBS Hostpath. +`./jeannie` reads `homelab.yml` at startup and exports matching OpenTofu +variables for the edge, provisioning, cluster, platform, and apps stacks. This +keeps values such as the Debian LAN IP, Debian Tailscale IP, RPi LAN IP, OCI IP, +Traefik MetalLB IP, and domain name from drifting across scripts and templates. +`scripts/render-docs` also renders [docs/service-catalog.md](docs/service-catalog.md) +and the static service catalog page at +`apps/demos-static/public/homelab-catalog/index.html` from the same inventory. + +Run `./jeannie up` and `./jeannie nuke` only from the Debian homelab server. The +script intentionally refuses to run from non-Debian machines so a laptop cannot +accidentally modify the cluster. + +## Flow + +1. `bootstrap/provisioning` + - prepares a Debian server as a PXE and preseed service for arm64 VMs + - serves Debian 13 arm64 netboot assets through TFTP and HTTP + - exposes a PXE rescue menu and HTTP rescue page for manual repair paths + - creates a golden image install path with Kubernetes, containerd, + qemu-guest-agent, cloud-init, and storage client packages ready + - is driven by `./jeannie up` when Pimox is reachable, without changing + Orange Pi host networking + +2. `bootstrap/cluster` + - creates the kubeadm control plane on the Debian amd64 node + - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes + - configures Calico-compatible pod CIDR + - configures containerd to pull from the in-cluster NodePort registry + - creates retained host directories under `/data/openebs/local` + +3. `bootstrap/platform` + - installs a minimal Calico deployment through the Tigera operator + - installs NodeLocal DNSCache for node-local DNS query caching + - installs MetalLB for LAN `LoadBalancer` services + - installs Traefik as the single Kubernetes ingress entry point + - installs OpenEBS + - creates `openebs-hostpath-retain` + - installs Argo CD + - installs Kyverno with audit-first baseline Pod Security policies and + homelab image signature/SBOM admission controls + - registers the private GitOps repo without storing the SSH private key in + Terraform state + +4. `bootstrap/apps` + - registers Argo CD Applications from the `applications` map + - passes the website image produced by the build step into Argo CD as a + Kustomize image override + - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM + listens on loopback inside the container, OPcache is enabled, and + pre-rendered HTML handles the read-heavy pages + - default apps are `container-registry`, `website-production`, + `demos-static`, the `heimdall` route bundle, `n8n`, and + `supply-chain-policy` + +5. `bootstrap/edge` + - connects to the OCI jump box + - uploads nginx, HAProxy, Varnish, and Squid configs + - obtains and renews Let's Encrypt certificates for the configured hostname + - runs the edge cache/proxy chain with Docker Compose + - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream + keepalive, and cache headers for the public website path + +## Prerequisites + +On the Debian host: + +- OpenTofu +- Docker with Buildx +- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes + minor, currently `v1.36` +- SSH access to worker nodes +- SSH access to the OCI edge host +- Go toolchain for Jeannie core helpers; `./jeannie up` installs + `golang-go` on the Debian host if it is missing +- enough persistent storage for `/data/openebs/local` and Docker's data root + +After a clean Debian reinstall, bootstrap the host package set and desktop +configuration first: + +```bash +sudo apt-get update +sudo apt-get install -y git ansible sudo +git clone /home/jv/my-homelab-configs +cd /home/jv/my-homelab-configs +ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml +``` + +If you kept the private app-config archive generated from the old Debian host, +pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. +Details are in `bootstrap/host/README.md`. + +The default kubeconfig path is `/home/jv/.kube/config`. Override it with +`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. + +## Implementation Language Policy + +Jeannie keeps `./jeannie` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./jeannie doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./jeannie ai-evals run +``` + +Use `./jeannie ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Flow + +1. `bootstrap/provisioning` + - prepares a Debian server as a PXE and preseed service for arm64 VMs + - serves Debian 13 arm64 netboot assets through TFTP and HTTP + - exposes a PXE rescue menu and HTTP rescue page for manual repair paths + - creates a golden image install path with Kubernetes, containerd, + qemu-guest-agent, cloud-init, and storage client packages ready + - is driven by `./jeannie up` when Pimox is reachable, without changing + Orange Pi host networking + +2. `bootstrap/cluster` + - creates the kubeadm control plane on the Debian amd64 node + - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes + - configures Calico-compatible pod CIDR + - configures containerd to pull from the in-cluster NodePort registry + - creates retained host directories under `/data/openebs/local` + +3. `bootstrap/platform` + - installs a minimal Calico deployment through the Tigera operator + - installs NodeLocal DNSCache for node-local DNS query caching + - installs MetalLB for LAN `LoadBalancer` services + - installs Traefik as the single Kubernetes ingress entry point + - installs OpenEBS + - creates `openebs-hostpath-retain` + - installs Argo CD + - installs Kyverno with audit-first baseline Pod Security policies and + homelab image signature/SBOM admission controls + - registers the private GitOps repo without storing the SSH private key in + Terraform state + +4. `bootstrap/apps` + - registers Argo CD Applications from the `applications` map + - passes the website image produced by the build step into Argo CD as a + Kustomize image override + - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM + listens on loopback inside the container, OPcache is enabled, and + pre-rendered HTML handles the read-heavy pages + - default apps are `container-registry`, `website-production`, + `demos-static`, the `heimdall` route bundle, `n8n`, and + `supply-chain-policy` + +5. `bootstrap/edge` + - connects to the OCI jump box + - uploads nginx, HAProxy, Varnish, and Squid configs + - obtains and renews Let's Encrypt certificates for the configured hostname + - runs the edge cache/proxy chain with Docker Compose + - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream + keepalive, and cache headers for the public website path + +## Prerequisites + +On the Debian host: + +- OpenTofu +- Docker with Buildx +- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes + minor, currently `v1.36` +- SSH access to worker nodes +- SSH access to the OCI edge host +- Go toolchain for Jeannie core helpers; `./jeannie up` installs + `golang-go` on the Debian host if it is missing +- enough persistent storage for `/data/openebs/local` and Docker's data root + +After a clean Debian reinstall, bootstrap the host package set and desktop +configuration first: + +```bash +sudo apt-get update +sudo apt-get install -y git ansible sudo +git clone /home/jv/my-homelab-configs +cd /home/jv/my-homelab-configs +ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml +``` + +If you kept the private app-config archive generated from the old Debian host, +pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. +Details are in `bootstrap/host/README.md`. + +The default kubeconfig path is `/home/jv/.kube/config`. Override it with +`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. + +## Implementation Language Policy + +Jeannie keeps `./jeannie` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./jeannie doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./jeannie ai-evals run +``` + +Use `./jeannie ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) -### 3. AI Integration Flow -AI capabilities are decoupled from the K8s lifecycle to utilize host-level GPU/CPU resources efficiently. -- **API Layer:** Ollama runs as a host service. -- **Consumption:** K8s-based applications communicate with the Ollama API over the internal LAN. -- **Memory:** Redis (transitioning to Valkey) provides a translation and session cache for AI-driven web features. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..e2c15d4 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,3605 @@ +# Homelab Deployment Guide + +## Prerequisites + +On the Debian host: + +- OpenTofu +- Docker with Buildx +- kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes + minor, currently `v1.36` +- SSH access to worker nodes +- SSH access to the OCI edge host +- Go toolchain for Jeannie core helpers; `./jeannie up` installs + `golang-go` on the Debian host if it is missing +- enough persistent storage for `/data/openebs/local` and Docker's data root + +After a clean Debian reinstall, bootstrap the host package set and desktop +configuration first: + +```bash +sudo apt-get update +sudo apt-get install -y git ansible sudo +git clone /home/jv/my-homelab-configs +cd /home/jv/my-homelab-configs +ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml +``` + +If you kept the private app-config archive generated from the old Debian host, +pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. +Details are in `bootstrap/host/README.md`. + +The default kubeconfig path is `/home/jv/.kube/config`. Override it with +`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. + +## Implementation Language Policy + +Jeannie keeps `./jeannie` as the stable command entrypoint. New durable +read-only orchestration, report rendering, inventory parsing, status/doctor +logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because +it ships as one small binary and is easier to test than Bash. Bash remains for +thin compatibility wrappers and remote shell glue. Python remains acceptable for +AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the +clear fit. New repo features should choose the smallest language that fits the +task instead of defaulting to Bash. + +## Version Discipline + +The lab should converge on one Kubernetes minor across the API server and every +kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers +match the Debian control plane and Raspberry Pi worker instead of staying on an +older template. Avoid treating a three-minor kubelet skew as normal steady +state: it is only a temporary compatibility window and blocks the next control +plane upgrade. + +Check node versions from the Debian host: + +```bash +./jeannie doctor-versions +``` + +The check prints the API server, kubelet versions, kubelet minors, and containerd +runtimes. It exits nonzero when kubelet minors differ from the API server minor +or when containerd major versions are mixed. The Gitea Actions auto path runs +this check before app deploys; kubelet minor drift switches the job to +`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced +from the pinned image. For Pimox workers, prefer rebuilding from the corrected +golden template and rejoining the worker over ad hoc live package upgrades. + +## AI Evals + +The repo includes a local eval harness for Jeannie explanations and homelab RAG +grounding. It uses fixtures and allowlisted read-only commands, so it is safe to +run before changing the lab: + +```bash +./jeannie ai-evals run +``` + +Use `./jeannie ai-evals list` to see the current cases. Add new cases +under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior +should become repeatable. + +## Prompt Injection Lab + +The defensive security training path includes local prompt-injection fixtures +for hostile runbooks, tool output, dashboard annotations, and command arguments. +Run it without touching live infrastructure: + +```bash +./jeannie prompt-injection-lab run +``` + +Cases live under `security/prompt-injection-lab` and are meant to become +regression tests for future Jeannie AI features. + +## Incident Commander + +The repo also includes a read-only incident commander trainer. It classifies +pasted failure text, prints a likely hypothesis, links the right runbook, and +lists the next read-only Jeannie commands: + +```bash +./jeannie incident replay +./jeannie incident triage --text "502 Bad Gateway from nginx" +``` + +Incident fixtures live under `infra/incident-commander`. Use them to turn +repeat outages into repeatable triage behavior before adding any self-healing. + +## Safety Cases + +Before high-blast-radius changes, generate a safety case from the Git diff: + +```bash +./jeannie safety-case --since HEAD~1 +``` + +It summarizes risk, blast radius, required validation, rollback, and forbidden +actions. The implementation lives under `infra/safety-cases` and reuses the +same impact rules as `./jeannie impact`. + +## Agent Sandbox + +Future Jeannie AI actions should be checked before execution. The local sandbox +policy starts with a read-only command classifier: + +```bash +./jeannie agent-sandbox check "./jeannie status" +./jeannie agent-sandbox check "./jeannie nuke" +``` + +Policy is stored in `infra/agent-sandbox/policy.tsv`. + +## AI Workload Scheduling + +For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox +workers, and the RPi based on CPU, memory, architecture, tailnet access, and +control-plane policy: + +```bash +./jeannie ai-scheduler recommend small-rag +./jeannie ai-scheduler simulate +``` + +The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a +real Kubernetes controller. + +## Memory With Provenance + +Jeannie can return cited retrieval context and check whether the local RAG index +is stale: + +```bash +./jeannie ask --citations "how do I recover the cluster?" +./jeannie ai-memory check +``` + +This keeps local AI answers grounded in repo files and runbooks. + +## Red/Blue Loop + +Safe red-team/blue-team checks are grouped under one command: + +```bash +./jeannie red-blue plan +./jeannie red-blue run --local +``` + +Local mode runs deterministic prompt-injection and eval checks. Live scans are +listed but gated so they can run from the Debian host when intended. + +## Canary Promotion + +The promotion helper defines release gates before production changes: + +```bash +./jeannie promote plan +./jeannie promote validate +./jeannie promote rollback website-production +``` + +The current version is intentionally conservative: it validates local gates and +prints rollback steps while dev namespace automation remains explicit future +work. + +## Guarded Self-Healing + +Jeannie can turn status findings into an experimental self-heal plan: + +```bash +./jeannie status --heal-plan +./jeannie status --heal +./jeannie heal apply --yes +``` + +Normal `status` remains read-only. The heal planner maps known findings to +guarded actions, chooses one highest-impact auto-eligible finding, checks the +proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that +one action. Re-run `./jeannie status` after every heal cycle so Jeannie +can reassess before touching the next issue. If a specific Pimox worker is +NotReady, heal targets that worker with `./jeannie workers restart N` +instead of repeating a broad cluster start. After an action runs, Jeannie waits +for the target condition to clear and reports `healed` or `still failing`. +Failed heal actions are placed on a short cooldown so Jeannie does not loop on +the same VM restart; the next plan escalates to diagnostics instead. +Destructive actions remain blocked. + +## Model Behavior Observatory + +Stable prompts track local model behavior, latency, refusal behavior, and +tool-correctness signals over time: + +```bash +./jeannie model-observe prompts +./jeannie model-observe run --offline +./jeannie model-observe report +``` + +Use `--live` only when Ollama is available. + +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Deploying + +From the Debian server: + +```bash +cd ~/my-homelab-configs +./jeannie up +``` + +The script first deploys external Gitea to the Debian host with Docker Compose +under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays +outside the Kubernetes rebuild blast radius. It then detects the +Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are +available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 +arm64 template, creates or reuses one worker VM clone, discovers the guest IP +through qemu-guest-agent, and passes that worker into the cluster layer. It then +applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the +local registry, builds the website and demos images when their source changed, +pushes them to the registry, recreates pods only after a new image is built, and +applies the edge stack. + +Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set +`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline +keeps the template on its configured `local` storage, creates new worker VM +clones on `opi5_ssd` by default when that Pimox storage is available, +checks that the Pimox bridge already exists, refuses `local` as worker clone +storage, and refuses to edit Orange Pi host networking. Keep durable data on +the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks +as rebuildable worker capacity. + +`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns +worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set +`LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker +must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID +`9010` and VMID `9011`. + +Pimox workers can join the tailnet during cluster worker bootstrap. Keep the +ephemeral or pre-authorized auth key outside Git, either in +`~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in +`LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: + +```bash +LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./jeannie up +``` + +For existing workers, prefer the Jeannie wrapper because it reads the generated +worker keys and exports the correct OpenTofu variables: + +```bash +./jeannie workers tailnet +``` + +When applying `bootstrap/cluster` directly, use the OpenTofu variable name +instead of the Jeannie `LAB_` wrapper: + +```bash +TF_VAR_worker_tailscale_enabled=true \ +tofu -chdir=bootstrap/cluster apply \ + -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" +``` + +Worker Tailscale defaults to `--accept-routes=false` because these VMs already +live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can +steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT +from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox +workers can reach tailnet-only services. Set +`LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume +tailnet-advertised routes that do not overlap the local LAN. Set +`LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through +`tailscale0` is not wanted. `./jeannie doctor-cluster` checks Pimox +worker tailnet egress against +`${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and +`${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. + +OpenWrt firewall VM automation is available as a standalone command because it +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` +already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM +SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, +imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores +the VM disk on the selected non-local Pimox storage. It leaves the VM stopped +and not enabled for host boot by default. It does not use the Debian Kubernetes +golden-node template for OpenWrt. + +The website and demos images default to `linux/amd64,linux/arm64` because both +deployments can schedule on either the Debian host or ARM workers. Override with +`WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally +restricting node placement. + +Build metadata is written under `.lab/` so repeat runs can skip the website +or demos image build when the source hash, platform, image reference, and +registry manifest still match. + +The website build is intentionally performed on the Debian homelab path and the +Gitea Actions runner, not on a laptop. The current image uses +`php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables +OPcache for immutable image deploys, and runs +`apps/website/render-static-pages.sh` during the image build. That render step +produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and +`/homelab-tree.php` in the static languages. Runtime PHP remains for write and +translation endpoints such as `save_idea.php`, `visitor_ideas.php`, +`translate.php`, and `save_lang.php`. + +Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the +service is already managed manually. The default Gitea target is +`jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, +and SSH port `32222`. + +Before continuing into Pimox, cluster, apps, or edge changes, `./jeannie up` +runs `./jeannie preflight`. The preflight verifies Gitea, Debian Docker +root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, +Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory +or host storage/networking: + +```bash +./jeannie preflight +``` + +If the Debian Docker root check fails after a reinstall, repair it before +running the full pipeline: + +```bash +./jeannie fix-debian-docker-root +``` + +When `./jeannie up` sees an existing tracked kubeadm control plane but +the Kubernetes API is down, it starts the saved runtime before applying cluster +changes. Use `./jeannie rebuild-cluster` only when you intentionally +want to destroy and recreate the cluster. + +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Validation + +Useful checks after a rebuild: + +```bash +export KUBECONFIG=/home/jv/.kube/config + +kubectl get nodes +kubectl top nodes +kubectl -n argocd get applications +kubectl -n container-registry get pods +kubectl -n website-production get pods -o wide +kubectl -n demos-static get pods -o wide + +ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' +ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' + +docker info --format '{{.DockerRootDir}}' +df -h / /data /data/openebs/local /data/docker +``` + +The website should be reached through the configured public hostname, not the raw +OCI IP address, because the Let's Encrypt certificate is issued for the +hostname. + +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie help +./jeannie status +./jeannie capacity +./jeannie capacity-advisor +./jeannie capacity-limits +./jeannie recover-plan +./jeannie recover-power --dry-run +./jeannie impact --since HEAD~1 +./jeannie review-last-change +./jeannie scorecard +./jeannie explain scorecard +./jeannie gitops-status +./jeannie cert-check +./jeannie grafana-dashboards list +./jeannie release-snapshot +./jeannie backup-status +./jeannie synthetic-checks +./jeannie resource-budget +./jeannie control-plane status +./jeannie artifact-cache status +./jeannie blockchain-devnet status +./jeannie blockchain-test +./jeannie blockchain-wallet instructions +./jeannie golden-ledger check +./jeannie route-inventory +./jeannie workers list +./jeannie change-journal list +./jeannie map +``` + +`help` prints the grouped command reference in pipeline order, from inventory +and bootstrap through cluster lifecycle, reports, observability, recovery, +access, apps, security, AI, and destructive commands. + +`status` uses the reusable report UI and defaults to showing only failures and +warnings so the terminal stays readable. Use `./jeannie status --all` +to show every check, `--details` for row details, `--json` for structured +output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ +Tailscale/HTTP dump. The cascade includes "what broke" signals such as +deployment readiness, pod restarts, node pressure, disk pressure, Traefik +5xx/404 log evidence, and recent Gitea errors. + +High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node +readiness, pod restart storms, unavailable deployments, node/PVC storage +pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape +coverage. Keep this set small so alerts remain actionable. + +Application egress is controlled with Kubernetes NetworkPolicies where the lab +owns the workload. The website namespace defaults to denied egress and allows +only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is +isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS +for controlled practice targets. + +`capacity` is the placement report for the hardware you already have. By +default it uses the compact report renderer and shows summarized Debian, +Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use +`./jeannie capacity --details`, `--json`, `--only problems`, or +`--verbose` for the older full capacity dump. + +`capacity-advisor` interprets capacity, resource-budget, and control-plane +placement signals to recommend whether to add Pimox VMs, taint Debian, or fix +resource definitions first. + +`capacity-limits` prints suggested Kubernetes `resources.requests` and +`resources.limits` YAML snippets for containers missing them. It uses +`infra/resource-recommendations.yml` as a conservative starting profile and does +not edit manifests. + +`recover-plan` prints the disaster recovery order and lightweight prerequisite +checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and +edge verification. + +`recover-power` runs the post-outage recovery sequence in dependency order: +Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and +edge/public checks. Use `recover-power --dry-run` to preview the sequence. + +`impact` explains what a change may touch before you apply it. Use +`./jeannie impact --since HEAD~1` for the last commit or pass paths +like `bootstrap/edge` and `apps/website`. + +`review-last-change` summarizes the last commit, runs impact analysis for the +touched files, and prints validation plus rollback hints. + +`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster +health, edge health, capacity pressure, security posture, public certificates, +inventory, and docs freshness into a compact pass/warn/fail report. + +`explain` is the read-only follow-up for warnings, failures, and corrective +output. It can run safe report commands itself, such as +`./jeannie explain scorecard`, `status`, `capacity`, or +`resource-budget`, and it can explain captured output with +`./jeannie explain output < saved-output.txt`. It lists what each +finding means, diagnostic commands, whether a Jeannie fix command already +exists, and risk/auto-fix safety. + +Report-oriented commands share the reusable UI module in `scripts/report-ui` +and renderer in `scripts/report-render`. Commands emit status TSV rows, and the +renderer prints compact grouped summaries by default, with details or JSON +available for future TUI/web output. Rows may include separate `fix`, `graph`, +and `query` fields so a warning can point to both the next `jeannie` command and +the matching Grafana panel or Explore query. + +`gitops-status` prints a focused Argo CD view: application sync and health, +out-of-sync or degraded apps, repository secret presence, recent Argo CD events, +and recent repo-server/application-controller errors. + +`grafana-dashboards` manages repo-provisioned Grafana dashboards under +`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions +them through the platform stack; `./jeannie grafana-dashboards apply` +rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional +blockchain devnet dashboard uses metrics from +`infra/blockchain-devnet/metrics/exporter.py`. + +`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea +HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. + +`release-snapshot` writes a timestamped pre-change report under the homelab +state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, +public URL status, and pointers to the latest Gitea/OpenTofu backups. + +`backup-status` reports freshness for Gitea backups, OpenTofu state backups, +restore drill reports, and repo-managed Pi-hole restore inputs. The main +`status` cascade includes this as a warning signal. + +`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, +Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The +registry push/pull smoke test is opt-in with +`LAB_SYNTHETIC_REGISTRY_PUSH=true`. + +`resource-budget` checks the report-only policy in `infra/resource-budgets.yml` +against Kubernetes pod requests/limits and Debian host disk/memory signals. Use +it before adding heavier workloads so the HP laptop remains usable. By default +it summarizes missing requests/limits by namespace; use +`./jeannie resource-budget --details` for every pod/container. + +`control-plane` shows or manages the Debian Kubernetes control-plane scheduling +taint. After Pimox workers are stable, use `./jeannie control-plane taint` +to keep normal app/data-plane pods off the Debian host while control-plane and +DaemonSet workloads continue using their normal tolerations. + +`artifact-cache` manages an optional Debian-host cache stack in +`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through +cache. Use `artifact-cache instructions` for client configuration snippets. + +`blockchain-devnet` manages an optional local Ethereum Anvil devnet under +`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability +learning only; never use real seed phrases or mainnet keys. + +`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts +with a simple `Counter` contract and ownership tests so there is a known-good +baseline before practicing vulnerable contracts. + +`blockchain-wallet` provides disposable dev-wallet generation, address +derivation, and message-signing practice. It is documented in +`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. + +`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the +reviewable record of template VMID, storage, OS release, Kubernetes pins, +runtime versions, and build metadata for Pimox worker golden images. + +`route-inventory` reports Kubernetes ingress hosts, paths, backend services, +TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and +missing monitors are easier to spot. + +`workers` provides named Kubernetes/Pimox worker lifecycle operations: +`list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and +`rebalance`. +Use `recreate-plan ` before replacing one broken Pimox worker so the +cluster drain, VM stop, and `./jeannie up` path stay ordered. + +`change-journal` lists local journal entries written before risky commands such +as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, +`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live +under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, +dirty file count, and command details. + +`map` prints a dependency map from the canonical inventory, covering public +entry, GitOps automation, cluster foundation, DNS, observability, and security +learning paths. `./jeannie map --dot` emits Graphviz DOT. + +Focused doctor commands run narrower read-only checks. They use the same +problems-first UI as `status`; add `--all`, `--details`, `--json`, or +`--verbose` when you need more context: + +```bash +./jeannie doctor-edge +./jeannie doctor-gitea +./jeannie doctor-rpi +./jeannie doctor-cluster +``` + +`doctor-cluster` also checks whether Pimox workers have Tailscale installed, +`tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet +probe service. + +## RPi Services + +The Raspberry Pi runs lightweight always-on services outside Kubernetes from +`infra/rpi-services`: + +- Pi-hole on DNS port `53` and web port `8081` +- Unbound as Pi-hole's first upstream resolver +- Uptime Kuma on port `3001` + +Deploy or refresh them from the Debian server with: + +```bash +./jeannie rpi-services +``` + +`./jeannie up` also runs this step unless +`LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is +expected under `/nvme-storage/docker` when that path is an active writable mount point. If +the NVMe is missing or resets, it rewrites Docker's `data-root` to +`/var/lib/docker` and restarts Docker so DNS can still come up from the +SD/root filesystem. + +Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can +continue resolving if the Unbound container stops. Repo-managed Pi-hole state +lives in `infra/rpi-services/adlists.txt`, +`infra/rpi-services/local-dns-records.txt`, +`infra/rpi-services/cname-records.txt`, and +`infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without +deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into +a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` +is not provided, the bootstrap generates one and keeps it in +`/opt/homelab-rpi-services/.env`. +Uptime Kuma monitor seeds live in +`infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first +Kuma user exists. +`./jeannie status` and `./jeannie doctor-rpi` test Pi-hole DNS, +direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and +whether Docker is currently using `/nvme-storage/docker` or has fallen back to +`/var/lib/docker`. + +## Adding Nodes + +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 +template and worker VM clones automatically when the Orange Pi storage path is +healthy. Defaults are intentionally tied to the observed host: Pimox SSH host +`192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two +8 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, +and KVM launch affinity `4-7` because this Pimox is pinned to Debian Bullseye +on an Orange Pi 5 Plus. Jeannie launches the generated KVM command under that +host CPU set so PSCI can bring both guest CPUs online; it removes legacy `args` +entries so native `qm` CPU topology is the only `-smp` argument. Set +`LAB_PIMOX_KVM_CPUSET=` to use normal `qm start` on a different host. The NVMe +that was previously attached to the Orange Pi now belongs to the HP Debian +laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox +datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and +override variables are in +`bootstrap/provisioning/README.md`. + +Worker indexes are stable. Index `1` maps to VMID `9010`, node name +`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and +so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while +allowing higher indexes to be automated. + +Worker networking defaults to static because the ISP modem DHCP server does not +answer Pimox worker VM DHCP requests behind `vmbr0`. Set +`LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM +traffic. Static mode uses +`LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, +`LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and +`LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. +For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions +for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` +otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before +adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` +only when the Pimox host firewall already permits bridged worker IP traffic. +Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under +`${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. +Worker NIC MAC addresses default to the VMID-derived deterministic format used +by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` +only when you explicitly want Pimox to generate the MAC. +Worker creation fails faster than template creation: static guest-network +configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, +worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, +and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. +Static guest-network validation skips the extra LAN ICMP probe by default +because the ISP modem may not answer worker VM pings. Set +`LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host +should be required during qemu-guest-agent network setup. After +qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by +default and lets SSH be the reachability check. Set +`LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. +`LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. +Before `./jeannie apps` changes any application resources, Jeannie +requires every generated Pimox worker to be `Ready` and expose at least two +Kubernetes CPUs. Override the threshold with +`LAB_PIMOX_APP_DEPLOY_MIN_CPUS`; set +`LAB_PIMOX_APP_DEPLOY_REQUIRE_CAPACITY=false` only for a deliberately +non-Pimox deployment. + +Long-running `./jeannie up` steps keep compact output by default, but +the active step line refreshes with elapsed time and the latest log line. Set +`JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or +`JEANNIE_VERBOSE=true` to stream full command output. +Independent host services run through the Go DAG runner during `up`: Gitea +deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks +can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential +execution while troubleshooting, or raise it above the default `4` if the lab +can absorb more concurrent work. + +Run a full cluster rebuild from the Debian server with: + +```bash +./jeannie rebuild-cluster +``` + +That path preserves external Gitea, rebuilds the Pimox template +with 2 cores and 8 GiB memory, replaces two Pimox worker VMs with 2 cores and +8 GiB memory, and joins those workers to the Kubernetes cluster. Pimox 7 ARM +does not support `qm --affinity`; Jeannie instead starts the generated KVM +command under CPU set `4-7` and verifies every worker reports two guest CPUs +before Kubernetes workloads proceed. +The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` +does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. + +To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, +PV data, or VM disks, run: + +```bash +./jeannie stop-cluster +``` + +This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when +available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully +shuts down automated Pimox worker VMs. It intentionally leaves the host +`containerd`/Docker stack running so external Docker Compose services such as +Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove +state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want +to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` +to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a +hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, +`LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to +select VM workers. Resume the runtime with: + +```bash +./jeannie start-cluster +``` + +Kubernetes worker generation defaults to automated Pimox workers only. The +Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` +is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless +`LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, +add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: + +```hcl +worker_nodes = { + raspberrypi = { + host = "192.168.100.89" + user = "jv" + node_name = "raspberry" + ssh_key_path = "/home/jv/.ssh/id_ed25519" + } +} +``` + +Stateful apps currently pin retained local PVs to the `debian` node. Move or +duplicate those PV manifests when you want storage on another node. + +## Workload Placement + +`bootstrap/cluster` labels nodes with homelab placement metadata: + +- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` + shows `worker` instead of `` in the ROLES column +- `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=control-plane` on the Debian control plane +- `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and + `homelab.dev/workload-class=edge` on the Raspberry Pi worker +- `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and + `homelab.dev/workload-class=platform` on automated Pimox worker clones when + those workers are enabled + +Override `control_plane_node_labels`, `worker_node_labels`, +`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when +the physical layout changes. The current website, demos, and registry manifests +are not moved automatically because the public NodePort path and retained +OpenEBS hostpath PVs are node-local. Move workloads only after their storage and +edge path are ready on the target node. Gitea is outside Kubernetes and is moved +by changing the Raspberry Pi Docker install target instead. + +The stateless platform controllers are pinned to Pimox worker nodes through +`homelab.dev/workload-class=platform` and include hostname topology spread plus +preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and +kube-state-metrics scheduling does not collapse onto the first worker that joins. +PVC-backed monitoring StatefulSets are intentionally treated separately because +their retained OpenEBS hostpath volumes are node-local. Run +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing +worker nodes, destroy only the existing `prometheus-stack` Helm release, delete +its retained PVC/PV objects, and recreate the stack on the worker selector when +you intentionally accept losing that monitoring data. A planned monitoring data +migration should be handled as a separate maintenance task with backup, +delete/recreate or storage migration steps, and post-restore checks. + +The older NodePort path is now reserved for special cases such as the local +registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` +support, but app traffic should normally enter through Traefik's MetalLB +address instead of per-app NodePorts. Before applying the edge stack, Jeannie +configures a Tailscale subnet router so the OCI edge can route to the Traefik +LoadBalancer address. The default route advertiser is the Debian control-plane +host because it can reach both Tailscale and the MetalLB address; set +`LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. + +The cluster stack still supports worker-based subnet-route advertisement when +that is useful: + +```hcl +metallb = { + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.240"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +} + +traefik = { + enabled = true + repository = "https://helm.traefik.io/traefik" + chart_version = "40.2.0" + namespace = "traefik" + load_balancer_ip = "192.168.100.240" + ingress_class = "traefik" +} + +tailscale_subnet_routes = { + enabled = true + worker_key = "raspberrypi" + routes = ["192.168.100.0/24"] +} +``` + +DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for +the service hostnames land on the same edge host. For direct LAN testing, point +LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at +Traefik's address: + +```text +192.168.100.240 lab2025.duckdns.org +192.168.100.240 demos.lab2025.duckdns.org +192.168.100.240 n8n.lab2025.duckdns.org +192.168.100.240 prowlarr.lab2025.duckdns.org +192.168.100.240 sonarr.lab2025.duckdns.org +192.168.100.240 radarr.lab2025.duckdns.org +192.168.100.240 qbittorrent.lab2025.duckdns.org +192.168.100.240 kapowarr.lab2025.duckdns.org +192.168.100.240 suwayomi.lab2025.duckdns.org +192.168.100.240 maintainerr.lab2025.duckdns.org +192.168.100.240 grafana.lab2025.duckdns.org +192.168.100.240 argocd.lab2025.duckdns.org +``` + +Heimdall is external to Kubernetes. For direct LAN testing, use +`http://192.168.100.73:8082/`; public access still uses +`https://heimdall.lab2025.duckdns.org/` through the OCI edge. + +The edge stack uses Traefik as its backend by default and validates +`http://192.168.100.240:80/` before updating the edge containers. If Tailscale +subnet-route approval is not automatic in the tailnet policy, the edge deploy +will fail clearly instead of silently keeping a broken public path. + +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +remote SSH targets when more worker nodes exist. Set it to an empty string for a +single-node rebuild. + +## Adding Platform Tools + +Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. + +## Policy Guardrails + +`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security +policies in `Audit` mode. This gives the lab policy reports for unsafe workload +settings without blocking existing pods during the first rollout. After reports +are clean, individual policies can be promoted to `Enforce` in +`bootstrap/platform/main.tf`. + +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.73:30500/php-website:*` and +`192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, +and audits whether the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, and downloaded Cosign binary live in +`${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing +identity survives temporary checkout cleanup. Repo-local image state files still +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses +the persistent Cosign key, publishes the public key into the Kyverno namespace as +`homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs +the pushed images, attaches a signed SPDX SBOM predicate, and verifies both +artifacts before refreshing the workload Applications. The policy starts in +`Audit` mode so an existing live deployment can recover while the current images +are signed; after `./jeannie apps` verifies both images, change `validationActions` in +`apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to +`Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to +manage the key password externally; otherwise the Debian runner creates +`cosign/cosign.password` under the homelab state directory for non-interactive +runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override +those paths. + +## DNS Cache + +`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with +`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on +`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the +rollout compatible with the current kube-proxy iptables path without rewriting +kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service +CIDR or upstream DNS servers change. + +## MetalLB + +MetalLB is present in `bootstrap/platform` but disabled by default. Enable it +only after reserving a LAN IP range outside DHCP and outside any future OpenWrt +LAN pool: + +```bash +export TF_VAR_metallb='{ + enabled = true + repository = "https://metallb.github.io/metallb" + version = "0.16.0" + namespace = "metallb-system" + address_pool = ["192.168.100.240-192.168.100.250"] + l2_advertisement_enabled = true + pool_name = "homelab-lan" +}' +``` + +Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services +such as the website, demos, and n8n should be `ClusterIP` services behind +Ingress objects. The local registry remains a `NodePort` because the cluster +nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; +they run on the Debian Docker host. + +## Secrets + +Use SOPS with age for secrets that need to live in Git. The Debian host +bootstrap installs `age` and `sops`; then run: + +```bash +./jeannie secrets-init +./jeannie secrets-check +``` + +`secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders +`.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. +Commit `.sops.yaml`, but keep the private age key outside the repo. Operational +notes are in `docs/secrets.md`. + +## Edge Services + +The OCI jump box runs the public edge path: + +```text +nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP +``` + +Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. +Validate it before applying with: + +```bash +./jeannie tailnet-policy-check +``` + +The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and +deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in +`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` +file when the public host, SSH key, server name, backend Tailscale IP, or +Traefik backend address changes. + +The `/git/` route is intentionally different from the Kubernetes app routes: it +proxies to Gitea on the Debian host through Tailscale instead of Traefik. This +keeps public read-only source browsing available even when the cluster has been +destroyed. + +For the main website, nginx terminates TLS, serves cached HTML and static +assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS +sessions, and keeps upstream connections open to the Traefik backend. The +application side marks pre-rendered HTML as +`Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS +under the stronger immutable asset policy. In local measurements, cached +single-resource requests are dominated by the remote TCP/TLS path, while a +reused HTTP/2 asset batch completes in roughly 60 ms after the connection is +established. + +Use the configured `server_name` in the browser, for example +`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser +certificate warning because the trusted certificate is issued for the hostname. + +The edge stack uses HTTP-01 validation and requests one certificate covering +`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains +under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive +expiry notices, or leave it empty to register without an email. Set +`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. + +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + +## Adding Apps + +Add Kubernetes manifests under `apps/` and register them in +`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and +self-healing for the app. + +The `heimdall` Argo CD app name is retained as a small route bundle for +Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can +prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes +workloads there. Use `./jeannie deploy-heimdall` for the dashboard +itself. + +The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is +exposed at `https://n8n.lab2025.duckdns.org`. It receives +`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing +Debian-host Ollama API for translation prewarming and batch jobs. + +The ARR/media stack is intentionally not deployed by the Kubernetes app +pipeline. It is better suited to a Docker Compose stack on the Debian host with +persistent volumes under `/data`, where large media paths, qBittorrent writes, +and service config directories can survive cluster rebuilds without OpenEBS +node-affinity or PVC migration concerns. The active host-level Compose +definition lives under `infra/arr-stack` and stores persistent data under +`/data/arr`. Historical Kubernetes manifests are archived under +`archive/apps/arr-stack` for migration reference only. + +## Storage + +OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use +retained local PV paths such as `/data/openebs/local/registry`; these paths are +intentionally outside kubeadm reset paths so data can survive cluster +destroy/create cycles. Those critical volumes are declared explicitly as +retained local PVs so a rebuilt cluster binds back to the same host paths +instead of creating fresh directories. + +For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the +standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath +data still lives under `/data/openebs/local`, and larger service data such as +Gitea, Ollama models, and app volumes should stay under `/data`. + +## Gitea + +Gitea is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a +Kubernetes workload. This keeps Git available when the Kubernetes cluster is +destroyed and rebuilt. + +The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. +Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. + +Public source browsing stays available through +`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users +can view public repositories, so the blog can link to code read-only while +writes still require an authenticated Gitea account. + +## Heimdall + +Heimdall is external bootstrap infrastructure. It runs on the Debian host as an +always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not +as a Kubernetes workload. This keeps the lab navigation page available when the +cluster is destroyed, rebuilding, or unhealthy. + +Deploy or refresh it from the Debian server with: + +```bash +./jeannie deploy-heimdall +``` + +`./jeannie up` also runs this step during the independent host +services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path +is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. +Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar +container that upserts repo-managed tiles without deleting manually added ones. + +The Debian bare repo remains the GitOps mirror: + +```text +/home/jv/git-server/my-homelab-configs.git +``` + +Argo CD consumes that Debian mirror through the default `gitops_repo_url`. +Gitea Actions pushes the `main` commit into the mirror before running the +selected deploy command. + +The platform bootstrap registers the Argo CD repository secret and the SSH host +key for the Debian GitOps mirror. If Argo CD reports +`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host +key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, +restart `argocd-repo-server`, and hard-refresh the affected Application. + +Deploy or refresh the external Gitea container from the Debian host with: + +```bash +./jeannie deploy-gitea +``` + +## Gitea Backups + +`./jeannie up` installs a Debian-host systemd timer named +`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured +Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the +dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default +retention is 30 days. + +The same install step also creates `homelab-gitea-restore-drill.timer`. The +monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it +to a temporary directory, records a report under +`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It +does not write into the live Gitea data directory. + +Run a manual backup from the Debian server with: + +```bash +./jeannie backup-gitea +``` + +Run the restore drill manually with: + +```bash +./jeannie drill-gitea-restore +./jeannie drill-restore +./jeannie drill-pihole-restore +``` + +`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed +config inputs, the latest OpenTofu state backup archive when present, and the +GitOps rebuild path. It does not replace running services or mutate Kubernetes. + +Useful checks: + +```bash +./jeannie backup-status +systemctl list-timers homelab-gitea-backup.timer +systemctl list-timers homelab-gitea-restore-drill.timer +sudo systemctl start homelab-gitea-backup.service +ls -lh /home/jv/backups/gitea +ls -lh /home/jv/backups/gitea-restore-drills +``` + +## Gitea Actions + +This repo includes a Gitea Actions workflow at +`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and +`main`, and deploys only from `main`. That keeps the promotion path simple: + +```text +dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync +``` + +The workflow targets a repository-scoped Debian host runner with the label +`homelab-debian`. + +Run the same validation locally before promoting: + +```bash +./jeannie validate +``` + +## Identity and access audit + +Run a read-only access inventory from the Debian server with: + +```bash +./jeannie access-audit +``` + +The audit reports configured SSH targets and key file modes, Git/Gitea remotes, +Tailscale ACL policy validation, current Kubernetes authorization, broad +cluster-admin bindings, automation service accounts, and Gitea runner status. +It does not create users, tokens, kubeconfigs, or keys. Use it to move toward +separate identities: + +- admin kubeconfig only on the Debian control host +- read-only kubeconfig for dashboards and audits +- separate service accounts for automation +- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge +- repo-managed Tailscale ACLs + +The read-only Kubernetes identity is managed in `apps/access-control`. After +that app syncs, generate a separate read-only kubeconfig on the Debian host: + +```bash +./jeannie kubeconfig-readonly +``` + +The workflow only blocks automatic deploy for external Gitea service +changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, +`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. + +Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` +on the runner to force one path. + +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key +with the Gitea repository and switches the Debian working copy's `gitea` remote +to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key +is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a +different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH +access unchanged. The Actions deploy job uses the checked-out Actions workspace +as the source commit, updates the first available persistent checkout from +`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or +`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the +Actions workspace. It does not need SSH read access back to Gitea. + +Enable Actions for the repository in Gitea, then create a repository-level runner +token from: + +```text +https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners +``` + +Register and start the Debian runner from the Debian server: + +```bash +cd ~/my-homelab-configs +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner +``` + +The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and +uses a host label instead of a Docker job container because deployment needs the +Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. + +The deployment job is non-interactive. User `jv` must be able to run `sudo -n +true` on the Debian host for deployment commands that require sudo. + +Useful checks: + +```bash +systemctl status homelab-gitea-runner.service +journalctl -u homelab-gitea-runner.service -n 100 --no-pager +``` + +## Renovate + +`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu +providers, Helm chart versions, and the pinned tools used by the Gitea Actions +workflow. Renovate should open reviewable update branches or PRs only; it must +not auto-merge infrastructure changes. Keep app-only dependency updates on the +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server +for platform or provisioning updates. + +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + +## Website App + +The website is a PHP app under `apps/website`. It includes a home page, CV page, +blog page, and demos page, plus a lightweight translation flow backed by Redis, +n8n, and Ollama. +Static language files live in `apps/website/lang`; `en.php` and `nah.php` are +curated source files, with the Nahuatl home page intentionally biased toward as +many Nahuatl words as possible while keeping technical terms understandable. +Unsupported browser languages use the same-origin `/translate.php` endpoint, +which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the +browser never calls the private Ollama IP directly. Redis caches per-string +translation results in the `website-production` namespace, while n8n is +available for translation prewarming and batch workflow jobs. The default model +is the custom `website-translator` Ollama model defined in +`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved +through `save_lang.php` on the website PVC, and `translate.php` emits structured +logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. + +Create or refresh the Ollama model on the Debian server before deploying a +website image that points at it: + +```bash +./jeannie website-translation-model +``` + +Ollama must also listen on the Debian host LAN address so Kubernetes pods on +other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: + +```bash +./jeannie website-ollama-listen +``` + +The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` +is true: it installs the service, stores models under `/data/ollama/models`, +binds the API to the configured LAN listener, and pulls the lightweight +`ai_gateway.model`. On an existing host, apply just this setup with: + +```bash +./jeannie ollama-setup +``` + +`jeannie` also has a backstage local AI helper for doctor commands. +It is not a separate CLI action; when `ai_gateway.enabled` is true in +`homelab.yml`, doctor commands try the configured Ollama model and silently +skip the helper if Ollama is down. The default lightweight model is: + +```bash +ollama pull qwen2.5:0.5b +``` + +Disable it for low-CPU sessions with: + +```bash +LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge +``` + +Build the local homelab knowledge index separately from the main deployment: + +```bash +./jeannie ai-index +./jeannie ai-check +./jeannie ask "how do I check edge to Gitea?" +``` + +The index is built from the non-secret source list in +`infra/ai/knowledge-sources.txt`, skips paths in +`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. +Doctor commands use it as extra context when it exists, but infrastructure +deployment does not depend on it. +Use `ask` as the runbook and command finder. It returns the closest indexed +docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after +retrieval. + +The CV page has two client-side presentation modes: + +- `Elegant`: dark, minimal, terminal-inspired styling with a square profile + image and light green console text. +- `Fancy`: centered circular profile image, cursive orbit text, and a + cursor-following portrait rotation effect. + +The Demos page is a catalog in the PHP website. The actual demo applications are +served from a separate `demos-static` artifact under `apps/demos-static` and are +published through the `demos-static` Argo CD application. Public traffic reaches +them through the edge path at `/demo-apps/`. + +`./jeannie up` builds and pushes two independent images: + +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD + as a Kustomize image override +- `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images + +The website manifest keeps the stable base image name `php-website:bootstrap`. +During bootstrap, `jeannie` hashes `apps/website`, builds +`/php-website:src-`, exports that exact reference through +`TF_VAR_website_image_ref`, and the Argo CD Application applies it through +Kustomize. This keeps the GitOps source generic while the deployed image remains +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. + +After `./jeannie apps`, the live deployment image should be a content-hash tag, +for example `192.168.100.73:30500/php-website:src-...`. If it still shows +`php-website:latest`, Argo CD has not rendered the current Application source. +Check the `website-production` Application source, sync status, and repository +access before restarting pods. + +The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs +private, browser-only image compression and conversion using native Canvas APIs. +Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled +to WebAssembly with a TypeScript UI so the codec work stays fast and still +avoids backend uploads. + +The demos are designed to be local-first so the current cluster can serve them +from any Linux app node without turning either pod into an application server. +The website pod serves the portfolio shell and the `demos-static` pod serves +static demo bundles; CPU-heavy work runs in the visitor's browser. Because the +deployments can run on either Debian or ARM workers, avoid bundling large ML +models, server-side WebSocket probes, or backend video transcoders into either +image. If those demos become production-grade, lazy load model assets in the +browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 +Plus. + +Current demo inventory: + +- Client-side media cruncher: image conversion/compression with Canvas; future + Rust/Wasm codec path for video. +- Internet quality visualizer: live Canvas graph for latency, jitter, and + stability using same-origin browser probes; a dedicated WebSocket echo endpoint + would be the production version. +- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and + local text-log filtering. +- Architecture simulator: click-driven load, crash, and auto-scale simulation. +- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB + conversions. +- Privacy-first redactor: local image redaction prototype; future + onnxruntime-web plus quantized YOLO or face model path. +- Local sentiment sandbox: lightweight local sentiment, keyword, and summary + prototype; future Transformers.js/ONNX path. +- Model drift simulator: visual MLOps playground for spikes, corrupted inputs, + and retraining. + +The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. +Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. + +Keep the `.terraform.lock.hcl` files committed. They pin provider selections and +make bootstrap behavior reproducible across nodes and rebuilds. +## Destructive Rebuilds + +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico +links, iptables rules, and local OpenTofu state. It does not delete retained data +under `/data/openebs/local`. + +For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH +targets. It defaults to an empty string so worker nodes are not cleaned unless +you explicitly include them. + diff --git a/scripts/render-docs b/scripts/render-docs index aedb23b..14cf33d 100755 --- a/scripts/render-docs +++ b/scripts/render-docs @@ -122,7 +122,7 @@ main() { # 2. Render Modular Docs for tmpl in "${TEMPLATES_DIR}"/*.tmpl; do [[ -e "$tmpl" ]] || continue - local filename=$(basename "$tmpl" .tmpl) + local filename=$(basename "$tmpl" .md.tmpl) local output_file="${DOCS_DIR}/${filename}.md" if [[ "${mode}" == "check" ]]; then